August 2003 - Libraries

RE: Libraries and hierarchies
by Simon Peyton-Jones 06 Aug '03

06 Aug '03

| > Either way, a value constructed by package coll-1.0 will be | > type-incompatible with functions in coll-2.0. The former will have | > original names "coll-1.0:Foo.T" while the latter will have | > "coll-2.0:Foo.T". Trying to provide transparent type upgrade is too | > hard. | that seems resonable. presumably to upgrade a binary package to use a | new library, i simply need to specify a new mapping in its grafting | file, and recompile the package. is that correct? Provided you have source, yes. The idea is that * a bunch of source files (perhaps comprising a package, but perhaps not) depend on one or more packages, and import modules from those packages using ordinary hierarchical module names * when compiling the source files, you graft into the tree some package implementations that match what the source files expect | > Manuel didn't like the fact that an import could mean "relative or | > absolute". But presumably we want to continue to write little programs | > with three modules A, B, Main, and have Main just say 'import A'. So | > presumably the current directory is always implicitly grafted into the | > module hierarchy at the root -- and that is all we need for making | > internal references within a package work out. | am i right in assuming that "current directory" refers to the directory | in which the file being compiled is located? e.g. | if module Main has a declartion "import M" | ghc A/B/Main.hs | will look for "A/B/M.hs" No. Our current idea is that "current directory" means "directory in which you started the compiler. So in your example, "import M" will look for M.hs, not A/B/M.hs. It's too confusing if the modules within a package change the names by which they refer to their siblings! But that's a design choice independent of all the others. Simon

1 0

RE: Libraries and hierarchies
by Simon Marlow 06 Aug '03

06 Aug '03

Ketil Z. Malde writes: > I presume that paths specified by -i (isn't it?) also will be grafted > into the root of the tree. > > How do you deal with a file "Foo.lhs" declaring "module Bar.Zot.Foo", > say, in the current directory? Is it legal to import it as "import > Foo"? Or "import Bar.Zot.Foo"? Both? A file Foo.lhs in the current directory should declare module Foo, anything else is an error. Within the current package (ie. the current set of modules that you're compiling, whether that's a program or a library), it can only be imported as module Foo. In another package, it could be imported as Bar.Zot.Foo, if the package containing Foo has been sited at Bar.Zot. > Is it really necessary and desirable to specify the whole path as part > of the module declaration? Under the proposed scheme, the module name in the module declaration would be relative to the root of the current package. Hope this helps! Cheers, Simon

1 0

Re: Text I/O library proposal, first draft
by Ashley Yakeley 06 Aug '03

06 Aug '03

At 2003-08-05 20:24, Ben Rudiak-Gould wrote: >1. We don't really need to work out a single recoder interface that's best >for every purpose. Well, something has to be in the library. Certainly I'd like to see pure (non-IO) encodings: if these are fast enough, and we have all the available ones, it might be appropriate to define all others in terms of that. Otherwise I suppose there could be a base interface for pure encodings and a base interface for IO-based encodings. This of course would be less desirable from a purist viewpoint. >2. There are regional encodings in use which are not stateless (JIS is the >only one I know of personally), so I think that a currentLocaleEncoding >function will have to return a stateful interface. OK >We have little choice but to use the C library's tables, >and the only interface to them is the C conversion functions. ... >4. Existing C conversion libraries (whose interfaces we can't change) >store their state in opaque data structures and provide no way to copy >that state. Which conversion library were you thinking of? This is true for iconv, which has a very simple interface, but apparently not for libc: see <http://www.gnu.org/manual/glibc-2.2.5/html_node/Keeping-the-state.html> -- Ashley Yakeley, Seattle WA

1 0

Re: Text I/O library proposal, first draft
by Ashley Yakeley 06 Aug '03

06 Aug '03

In article <Pine.LNX.4.21.0307311506290.31823-100000(a)dark.darkweb.com>, Ben Rudiak-Gould <benrg(a)dark.darkweb.com> wrote: > [Crossposted to Haskell and Libraries. Replies to Libraries.] There's a Haskell Internationalistion mailing list too. Also check out the project on SF: <http://sourceforge.net/projects/haskell-i18n/> There's a bunch of my code for Unicode properties, plus a couple of UTF8 implementations. > module System.TextIOFirstDraft (...) where could be put in Text.* hierarchy > type BlockRecoder from to = > Ptr from -> BlockLength -> Ptr to -> BlockLength > -> IO (BlockLength,BlockLength) UArray and MArray would be slightly cleaner if you're doing the IO thing. But actually my biggest problem is that this is in the IO monad. Given your code, I should be able to write these without resorting to unsafePerformIO: encodeUTF8 :: String -> [Word8] decodeUTF8 :: [Word8] -> Maybe String -- Nothing if not valid Actually, if one makes certain assumptions about encodings, you could get away with something like this: type Encoder base t = t -> [base] type Decoder base t = forall m. (Monad m) => m base -> m t Is this any less efficient? Probably not if you're writing your BlockRecoders in Haskell. > type TextEncoder = BlockRecoder Word32 Octet > type TextDecoder = BlockRecoder Octet Word32 On GHC, Char has exactly the range 0 to 0x10FFFF, as per Unicode codepoints. If this becomes standardised as part of an internationalisation effort, you might want to use Char rather than Word32. -- Ashley Yakeley, Seattle WA

2 2

Test
by Ashley Yakeley 06 Aug '03

06 Aug '03

I've been having trouble posting to this list (but not the other Haskell-related lists) from gmane (NNTP gateway for mailing lists, see <http://gmane.org/> or connect to <nntp://news.gmane.org>). The two messages from me that turned up on the list out of order were messages I had to resend manually to the list address because gmane apparently dropped them. But if you're reading this, the problem is fixed. -- Ashley Yakeley, Seattle WA

1 0

RE: Libraries and hierarchies
by Simon Marlow 05 Aug '03

05 Aug '03

Hi Iavor, > i like this, as i've wanted something like this, and it also seems > simple, although i may be missing something. > if i download a package, i simply put it on my system somewhere, > and add this place to the compiler search path (via a flag, or a > compiler configuration flag). then we can have relative and absolute > imports: absolute imports would be searched for in the > compiler search > path, while absolute imports will be searched for in the same > directory > as the importing module. alternatively one could only have > one kind of > import, that is always searched for in the compiler path. if > i want to > use one version of a library or another, i simply need to adjust the > path when i compile my program. thus "import A.B" would mean i want > module B that is in a subdirectory A of one of the directories in the > path. ambiguous imports should be reported as errors. Since this whole topic is rather hard to think about (I constantly find myself thinking in circles, anyway), could I ask you to elaborate on the scheme you had in mind? - what do you write in the source? are any syntax extensions required? - module header - import declarations - what is the compiler's strategy for finding imports? Is it the same for source files and interface/object files? - what is distributed with a library (a) in source form, (b) in binary form? - can a compiler produce objects/interfaces that are not fixed to a particular absolute module name in the hierarchy? (i.e. so that we can do binary distributions of libraries that aren't tied to particular absolute module names). - is it possible to have modules that reside at several places in the hierarchy simultaneously? Cheers, Simon

2 1

RE: Text I/O library proposal, first draft
by Hal Daume 05 Aug '03

05 Aug '03

this is true; however, it is likely that it would be much faster for me to read in word8s and parse them to Ints myself than to read in Chars (using the old libraries) and parse those to ints (I've done some experiments and the overhead for reading haskell Chars is very non-negligible). i'd like to take advantage of this new infrastructure, if possible... -- Hal Daume III | hdaume(a)isi.edu "Arrest this man, he talks in maths." | www.isi.edu/~hdaume > -----Original Message----- > From: libraries-admin(a)haskell.org > [mailto:libraries-admin@haskell.org] On Behalf Of John Meacham > Sent: Thursday, July 31, 2003 4:36 PM > To: libraries(a)haskell.org; haskell(a)haskell.org > Subject: Re: Text I/O library proposal, first draft > > > presumably if you are doing random access on the file, it is > in a known > nonarbitrary text encoding (like utf8). in which case you can > read/access the file with the binary routines and just use the > appropriate text conversions to get data out. > John > > On Thu, Jul 31, 2003 at 03:55:44PM -0700, Hal Daume wrote: > > Hi Ben, > > > > > Bad things: > > > > > > * There's no way to implement fgetpos/fsetpos type > functionality, > > > because coders don't expose their internal state. (In > fact, there > > > would need to be a way to explicitly copy the state, > since it may > > > well include IORefs, Ptrs, etc.) Is this a serious problem? > > > > Yes! This is an enormously serious problem. At least for me. > > > > It's not a problem for writing files, but I really really > really need > > this functionality when reading files. Reason: I'm often > tooling around > > in very large (1gb or greater) files which happen to be > sorted on some > > sort of index and I need to do binary search in them. To > load all the > > file into Haskell or to do linear search is impossible. > > > > Other than that, I rather like the design. > > > > - Hal > > -- > > Haskell mailing list > > Haskell(a)haskell.org > > http://www.haskell.org/mailman/listinfo/haskell > > > > -- > -------------------------------------------------------------- > ------------- > John Meacham - California Institute of Technology, Alum. - > john(a)foo.net > -------------------------------------------------------------- > ------------- > _______________________________________________ > Libraries mailing list > Libraries(a)haskell.org > http://www.haskell.org/mailman/listinfo/libraries >

1 0

RE: Libraries and hierarchies
by Simon Marlow 05 Aug '03

05 Aug '03

Hi Alastair, Thanks for the comments. > I wonder if the extra complication is really that high. > Suppose that every module somehow has a unique name > (128 bit hash, global registry, whatever) and that we have > setup two mappings to the same thing: > > Foo.Bar -> Reid.Consulting.Bar.version27 > Baz.Bar -> Reid.Consulting.Bar.version27 > > to implement this in Hugs, it is _almost_ enough to simply > run all source code through a preprocessor which replaces > occurences of Foo.Bar and Baz.Bar with Reid.Consulting.Bar.version27. Yes, this is a simpler implementation. > >(b) needing unique package names, > > The original proposal tied this feature strongly to packages. > I wonder if this is really necessary. Perhaps not - we only use packages as a convenient notion for a collection of modules within which relative module names can be used, and as a way to identify the collection. > Putting that aside, it seems that unique package names are > fairly easy to come by. Some possibilities are: > > 1) Use the URI of the primary download site. > From choice, the URI would be for a tarfile or whatever > but all that really matters is that it is unique. > This will typically include a version number but, if not, > the user could add one if they care. > > 2) In the likely event that we have a few big repositories > and many small ones, simply prefix the name of the package > by the repository name or, if not available, name of author. > This is effectively the same as (1) but avoids using URIs. > > 3) Directory name on your system. We can't use (3) in GHC, because the package name has to be fixed at compile time, and can't be changed once the binaries have been generated. It's an advantage to have short package names, because (at least in GHC) the package name will be included in every symbol in a compiled library. URIs are attractive, but long. It will make debugging really painful. Another scheme would be to use short package names (eg. "gtk-0.15", "hgl-3.00"), and have a web page (eg. on the Wiki) where everyone can register package names, with links to the appropriate download sites. This works well for the Python folks, and I think having short, readable package names will make the whole scheme more accessible. --- Some notes on dependencies, which occurred to Simon & I: An interesting property of this scheme is that if you distribute a binary library, then you only have to list the package names on which it depends: it doesn't matter where in the hierarchy the packages are sited on the target machine, it will still work. If you distribute source code, then the dependency list must include both package names and sites; this is something that can easily be incorporated into Isaac's distribution toolkit. We need an easy way to set up the system with the correct packages & sites for compiling some code, but that shouldn't be too hard. Interestingly, this is another way that source code can unambiguously refer to libraries: by package name, site, and module name (with the former two properties being specified out-of-band in some additional matter that comes with the source code - perhaps a package specification or similar). This is rather less ugly than using GUIDs. We can also provide GUIDs, of course, because doing so is cheap. Cheers, Simon

1 0

RE: Libraries and hierarchies
by Simon Marlow 04 Aug '03

04 Aug '03

> > Modules in other packages can be imported only by uttering > their full > > path names in the global hierarchy (of the compiler that is > compiling > > the package). > i don't quite understand this part. how is a module going to import > modules from another package, if it does not know where these > packages > are going to be installed? if i am writing a module M, which imports > A.B.X.N which is module X.N from a package P that was > installed at A.B, > doesn't that force everyone who is trying to use my package > to install P at the same place: A.B? Well, this is the key issue. Standing back a moment, I see three alternatives to organising the hierarchy: 1. We have a global registry for module names. The hierarchy is always populated with the same modules on every Haskell system. We have a mechanism for naming libraries after email addresses or domain names, or GUIDs, to ensure that everyone can get a non-clashing module name when they want one. This suffers from (a) needing a global registry for ordinary names (i.e. the non-GUID ones) and (b) overly long module names, (c) no library versioning. 2. We have no global registry for module names. Everyone is free to name library modules whatever they choose. In practice, groups of Haskell programmers will get together and maintain collections of non-clashing libraries, such as the fptools/libraries collection. This suffers from (a) possible name clashes between libraries, (b) no way to unambiguously refer to a specifiy library in Haskell source, (c) still no way to abbreviate long module names, and (d) still no GUIDs or versioning. 3. As (2), but the full name of a module is not fixed in the source code, and can be mapped into the hierarchy at multiple places. This is the method proposed in the message that started this thread. Because modules can be placed in the hierarchy in multiple places, it gives a good way to add library versioning and GUIDs to libraries. This suffers from (a) extra complication in the implementation and specification of the language, (b) needing unique package names, (c) referring unambiguously to a specific library must be done by GUID. So I guess what we're saying is that we're proposing (3) as a solution to the problems of (2). It probably isn't the best solution - indeed, I'm still worried about the implementation difficulties and the fact that we need unique package names, but perhaps solutions to these can be found. To answer the question, to unambiguously refer to another library under this story, you must import it by GUID. Cheers, Simon

4 3

RE: Libraries and hierarchies
by Simon Marlow 04 Aug '03

04 Aug '03

> It also gives the advantage that modules can be moved in the > hierarchy (by a > command-line flag or whatever) specifically to support > whatever a particular > program happens to require. > > Would the following be possible under your proposal: > > module M1.A imports Collections.Foo and only works with version 1 > module M2.B imports Collections.Foo and only works with version 2 > module C imports M1.A and M2.B > > Could we compile/use M1.A with Graphics.UI mapped to > Graphics.UI.v1 and M2.B > with Graphics.UI mapped to Graphics.UI.v2 and then correctly > compile module C? Yes, absolutely. There are issues about exactly what the programmer would have to do to make this happen: under the story I had in mind you would have to fiddle with a user-local package.conf file to change the mapping. Perhaps command-line flags could be added to make this more convenient, though. Cheers, Simon

2 1