February 2004 - Libraries

Code for conversion to and from UTF8
by George Russell 13 Feb '04

13 Feb '04

I don't know if anyone else has done this, but I thought my own effort (attached) might be of interest. It converts String -> String to and from UTF8 representations. It /should/ compile with any Haskell98 compiler which also has the (base) library Data.Bits. I'd be happy to see it in the standard hierarchy, if that's desired. Good points: (1) I think I've tested it fairly thoroughly; (2) when decoding UTF8 sequences, it rejects all ill-formatted sequences, and such errors can be easily detected (without exception handlers). Bad points: it's unoptimised. {- This module contains functions for converting to and from the UTF8 representations for Strings. -} module UTF8( toUTF8, -- :: String -> String -- Converts a String (whose characters must all have codes <2^31) into -- its UTF8 representation. fromUTF8WE, -- :: Monad m => String -> m String -- Converts a UTF8 representation of a String back into the String, -- catching all possible format errors. -- -- Example: With the Haskell module Control.Monad.Error, you can -- instance this as -- (fromUTF8WE :: String -> Either String String) -- to get a conversion function which either succeeds (Right) or -- returns an error message (Left). ) where import Char import List import Data.Bits -- -------------------------------------------------------------------------- -- Encoding -- -------------------------------------------------------------------------- -- | Converts a String into its UTF8 representation. toUTF8 :: String -> String toUTF8 [] = [] toUTF8 (x:xs) = let xs1 = toUTF8 xs ox = ord x mkUTF8 :: Int -> String -> Int -> Int -> String mkUTF8 x0 xs0 xmask0 xmax0 = let xbot = 0x80 .|. (x0 .&. 0x3f) x1 = x0 `shiftR` 6 xs1 = chr xbot : xs0 in if x1 < xmax0 then chr (xmask0 .|. x1) : xs1 else let xmask1 = xmask0 .|. xmax0 xmax1 = xmax0 `shiftR` 1 in mkUTF8 x1 xs1 xmask1 xmax1 in if ox <= 0x7f then x : xs1 else if ox `shiftR` 31 /= 0 then error ("Huge character with code " ++ show ox ++ " detected in string being converted to UTF8.") else mkUTF8 ox xs1 0xc0 0x20 -- | Converts a UTF8 representation of a String back into the String, -- catching all possible format errors. -- -- Example: With the Haskell module Control.Monad.Error, you can -- instance this as -- (fromUTF8WE :: String -> Either String String) -- to get a conversion function which either succeeds (Right) or -- returns an error message (Left). fromUTF8WE :: Monad m => String -> m String fromUTF8WE [] = return [] fromUTF8WE (x0 : xs0) = let ox = ord x0 in case topZero8 ox of 7 -> do xs1 <- fromUTF8WE xs0 return (x0 : xs1) 6 -> fail "UTF8 escape sequence starts 10xxxxxx" 0 -> fail "UTF8 escape sequence starts 11111110" -1 -> fail "UTF8 escape sequence starts 11111111" n -> let r = 6 - n -- number of 6-bit pieces xtop = ox .&. ones n minx = bit ( if r == 1 then 7 else 5*r + 1 ) mkx [] _ _ = fail "UTF8 string ends in middle of escape sequence" mkx (ch : xs1) x0 count0 = do let och = ord ch if och .&. 0x80 /= 0x80 then fail ("UTF8 escape sequence contains continuing " ++ "character not of form 10xxxxxx") else return () let xbot = och .&. 0x3f x1 = (x0 `shiftL` 6) .|. xbot count1 = count0 - 1 if count1 == 0 then return (x1,xs1) else mkx xs1 x1 count1 in do (x,xs1) <- mkx xs0 xtop r if x < minx then fail ("UTF8 escape sequence contains character not " ++ "optimally encoded") else do xs2 <- fromUTF8WE xs1 return (chr x : xs2) -- -------------------------------------------------------------------------- -- Binary utilities -- -------------------------------------------------------------------------- -- | return the number of the top bit which is zero, or -1 if they -- are all zero, for a number between 0 and 255. topZero8 :: Int -> Int topZero8 i = case (findIndex not (map (\ bn -> testBit i bn) [7,6..0] )) of Just n -> 7 - n Nothing -> -1 -- | (ones i) is number with binary representation 1 written i times. ones :: Int -> Int ones i = bit i - 1

2 1

RE: [Haskell] Weaving the Web with Haskell
by Simon Marlow 05 Feb '04

05 Feb '04

> On a tangentially related point, portability of several of > the existing > standard hierarchical libraries is rather patchy too. For instance, > the main reason that nhc98 does not currently ship with System.Time > from the base package, or any of the network or unix packages, is that > the checked-in implementations have a tool-dependency on hsc2hs. > (The latter does not appear to have a stand-alone distribution > independent of ghc, so it would be necessary to incorporate > it into the > nhc98 tree, and I don't currently have the time available to untangle > hsc2hs's configuration and build system and re-tangle it for nhc98.) > > I guess I'm with Sigbjorn on wanting to reduce the number of tool > dependencies, even though I happen to like GreenCard and hsc2hs > for what they can automate. Or perhaps, rather than reducing the > dependencies, I want to reduce their impact, by making the tools much > easier to get hold of, configure, and install. Splitting off hsc2hs into a standalone distribution wouldn't be too difficult, and is definitely a worthwhile task if anyone would like to take it on. Looking at the sources, it has some #ifdef __NHC__ so it looks like it worked on nhc98 at some point. It's a single module Haskell program with an auxiliary file (template-hsc.h). It needs to know one or two things about the system: what the C compiler is called, or where GHC can be found. That's about it, as far as I can tell. Cheers, Simon

1 0

Patch for Debug.QuickCheck
by Martin Sjögren 05 Feb '04

05 Feb '04

I just noticed that while the check function and the Config type are exported from Debug.QuickCheck, there is no "default" config exported. I attached a patch that exports a 'defaultConfig' variable that's equal to the normal 'quick' config. /Martin

2 1

RE: [Haskell] Weaving the Web with Haskell
by Graham Klyne 05 Feb '04

05 Feb '04

At 16:23 03/02/04 +0000, Simon Peyton-Jones wrote: >Having solid libraries for XML and HTTP would be a jolly good thing. As >you know, there is plenty of raw material around, but it obviously >doesn't do what you want, yet. I think I'm very close, based on the HXml Toolkit software and a few bits. I was parsing XML directly from the web earlier today, using Hugs on Win32. >If you, together with other interested parties, were to get together to >do this, the rest of us would be very grateful. You could do that by >directly working on the libraries distributed with GHC and Hugs (we >could give you CVS access). I'd like to see this become part of the standard libraries, when it becomes sufficiently stable and portable. I can't say how Uwe Schmidt feels about this. > Or you could make a separate project of it, >and distribute the stuff separately, which would free you from the GHC >and Hugs's release cycles. If the new library infrastructure project takes off, that could be an attractive option. I assume that in due course, the hierarchical libraries would move into such a framework. My goal is to be able to write software that uses an XML library that is easy for people not experienced with Haskell to obtain and load into their running environment. #g ------------ Graham Klyne For email: http://www.ninebynine.org/#Contact

2 1

RE: [Haskell] Weaving the Web with Haskell
by Simon Peyton-Jones 04 Feb '04

04 Feb '04

Graham Having solid libraries for XML and HTTP would be a jolly good thing. As you know, there is plenty of raw material around, but it obviously doesn't do what you want, yet. If you, together with other interested parties, were to get together to do this, the rest of us would be very grateful. You could do that by directly working on the libraries distributed with GHC and Hugs (we could give you CVS access). Or you could make a separate project of it, and distribute the stuff separately, which would free you from the GHC and Hugs's release cycles. Simon | -----Original Message----- | From: haskell-bounces(a)haskell.org [mailto:haskell-bounces@haskell.org] On Behalf Of Graham Klyne | Sent: 28 January 2004 12:37 | To: Haskell Mailing List | Subject: [Haskell] Weaving the Web with Haskell | | For the past week or so, I've been wrestling with various bits of Haskell | libraries trying to get an XML parser running under Windows. Each time I | think I've resolved a problem, another pops up to take its place. So I | felt it was a good time to stand back and review my goals and approach. | | | Background | | I believe that Haskell has a number of characteristics that make it | eminently suitable for prototyping and deploying a range of web | technologies. My own interest is in the Semantic Web area, but I think | Haskell could also be usefully applied to Web Services and other web | applications, and (as a language) potentially stands up well against the | likes of Java and Python. The Web is spawning a host of little (and | not-so-little) languages, and Haskell seems to have the right features to | handle these in a clean, principled fashion. This is my working premise. | | Contemporary web applications are largely based on XML and HTTP. So a | prerequisite for widespread use of any language for Web applications is | good XML and HTTP support. Given the right libraries, Haskell could make | handling these aspects very straightforward. But there seems to be a lack. | | | Requirements | | Ideally, I think that the following requirements should be satisfied: | - full XML parser supporting all features of the current XML core and | namespace specifications. | - full HTTP support, including content negotiation, redirection, | read/write, GET, HEAD, PUT, POST, etc. | - available across all Haskell compilers on all platforms | | Practically, I think the following would be enough to make a big | difference, and this is what I'm currently looking for: | - XML support, not necessary validating, but accepting full XML syntax and | supporting including internal entity definitions and substitution and XML | namespaces. | - capability to access the representation of a Web Resource using HTTP GET | - supported uniformly by GHC and Hugs, on Linux and Windows platforms. | | | Where I am | | I have been trying to use the HXML toolbox, because I understand it's the | only XML parser for Haskell that supports XML namespaces. Unfortunately, | it seems to be rather dependent on older versions of GHC (unless I'm | missing something), which is making it more problematic to adopt than I had | hoped. Other packages, such as HaXml, seem to be more portable but as far | as I'm aware are missing key functionality (notably XML namespace support). | | | Looking forward | | I shall plug away for a little while with HXML toolbox -- currently I'm | stuck for an MD5 library module -- and it may be that, with a little help, | I'll win through: if I get it running with Hugs under Windows, I'll | consider that to be significant progress. | | But maybe I'm missing a trick: is there other software I should be | considering to satisfy my goals? Constructive suggestions would be welcome. | | I'm anticipating that as the library infrastructure project settles, we'll | see some effort to move more of this kind of support code into a common and | generally usable function library. I hope I shall be able to contribute to | such an outcome. | | I suggest that the XML and HTTP support I've mentioned would be a key | enabler to allow Haskell to be more widely used for real-world projects as | well as academic research into language design issues (which currently | seems to be its strongest developer community interest). | | #g | | | ------------ | Graham Klyne | For email: | http://www.ninebynine.org/#Contact | | _______________________________________________ | Haskell mailing list | Haskell(a)haskell.org | http://www.haskell.org/mailman/listinfo/haskell

3 2

RE: Win32 libraries for Haskell
by Simon Marlow 04 Feb '04

04 Feb '04

> > The situation seems to be this: > > > > * Sigbjorn is actively maintaining hslibs/win32 for Galois > reasons, and > > is > > reluctant to transfer to libraries/win32 because it's > untested (I'm > > assuming > > that the Green-Card dependency is soluble by putting the > GC output > > in the CVS tree). Fair enough. > > > > I have no motivation other than it is a useful library used by many. > > GreenCard output, and output by the FFI backend in particular, > is pretty clunky(*), so much so that the *.hs files in > hslibs/win32 has been > cleaned up quite a bit. Furthermore, GreenCard produces non-reentrant > code, which is why I don't recommend it being used, libraries/Win32 or > otherwise. > > hslibs/win32/ doesn't improve on the non-reentrancy problem, btw. > > --sigbjorn > > * - having written large chunks of the current GC codebase, I feel > entitled to criticize it :) Interesting - in what sense is the greencard output non-reentrant? Do you mean that it won't work in a multithreaded app, or is it a restriction on what you can do in a callback? Cheers, Simon

1 0

Re: Win32 libraries for Haskell
by Graham Klyne 04 Feb '04

04 Feb '04

At 11:15 03/02/04 +0000, Simon Peyton-Jones wrote: >I'm not sure how to make progress here. Is anyone else interested or >willing to take ownership of libraries/win32? (Graham? Krasimir?) >This isn't a GHC specific thing -- I believe/hope the same libraries >should work equally well for Hugs. Hmmm... I took a peek at the Library/Win32 stuff in the GHC source distribution, and it looks as if there may a steep-ish learning curve. While I'm prepared in principle to help where I can, I'm concerned about over-commitment... >The situation seems to be this: > >* Sigbjorn is actively maintaining hslibs/win32 for Galois reasons, and >is > reluctant to transfer to libraries/win32 because it's untested (I'm >assuming > that the Green-Card dependency is soluble by putting the GC output > in the CVS tree). Fair enough. That might remove one learning-curve dependency... are there disadvantages to doing this? I also noted the lack of a test suite: is there one for hslibs that might be migrated? #g ------------ Graham Klyne For email: http://www.ninebynine.org/#Contact

2 3

Spawning a process using Hugs/Win32
by Graham Klyne 04 Feb '04

04 Feb '04

I thought I might have a version of POpen that works with Win32/Hugs, but I'm only slightly successful. The function of POpen is to spawn a sub-process using the Win32 equivalent of pipes to supply standard input and to accept standard output and standard error from the spawned process. My starting point was a module called Win32Spawn, which I found in hslibs/win32 in the GHC source distribution. I have managed to compile the accompanying module and FFI interface into a DLL. The files used to do this are at [1], with the build process outlined in [2]. The key code in the new module POPen.hs [3] is in function "spawn", which was adapted from the original module "hslibs/win32/Win32Spawn.hs". I had to change the code for "spawn", because the Hugs interface for function openFd is has fewer parameters than that provided by GHC. The remaining parameters seem to be the same as for the GHC version of openFd. The resulting program has compiled successfully, and the spawned program is activated as requested (in this case, curl: my evidence is that my personal firewall is asking me if curl should be allowed to access the Internet). But I also get an I/O exception in the calling program, in function spawn, when trying to open a handle to write the spawned process' standard input stream. My evidence for this is by this modification to the code: [[ hIn <- catch (openFd (fromIntegral wIn) False WriteMode False) (\er -> do { putStrLn ("openFd wIn: "++show er) ; ioError er } ) ]] generating this message: [[ openFd wIn: openFd: does not exist (file does not exist) ]] My question is this: are there any fundamental differences between GHC and Hugs, either in the implementation of openFd, or in the FFI interface, that would cause (what I assume is) a divergence of behaviour? Does anyone have any other clues what might be going wrong here? (BTW, I did get the HXml Toolbox to read and parse XML *from the web* using the internal HTTP library.) #g -- [1] http://www.ninebynine.org/Software/HaskellUtils/Win32/ [2] http://www.ninebynine.org/Software/HaskellUtils/Win32/MakePOpenDLL.bat [3] http://www.ninebynine.org/Software/HaskellUtils/Win32/POpen.hs [4] http://www.ninebynine.org/Software/HaskellUtils/Win32/Win32Spawn.hs ------------ Graham Klyne For email: http://www.ninebynine.org/#Contact

1 0

Win32 libraries for Haskell
by Simon Peyton-Jones 04 Feb '04

04 Feb '04

The situation seems to be this: * Sigbjorn is actively maintaining hslibs/win32 for Galois reasons, and is reluctant to transfer to libraries/win32 because it's untested (I'm assuming that the Green-Card dependency is soluble by putting the GC output in the CVS tree). Fair enough. * Alastair believes that lilbraries/win32 should work, but cannot maintain it. Fair enough. * Everyone else would far rather not to have to choose * The sooner we can dump hslibs/ the better I'm not sure how to make progress here. Is anyone else interested or willing to take ownership of libraries/win32? (Graham? Krasimir?) This isn't a GHC specific thing -- I believe/hope the same libraries should work equally well for Hugs. Simon | -----Original Message----- | From: cvs-all-bounces(a)haskell.org [mailto:cvs-all-bounces@haskell.org] On Behalf Of Sigbjorn Finne | Sent: 29 January 2004 02:37 | To: Simon Peyton-Jones | Cc: cvs-ghc(a)haskell.org; Graham Klyne | Subject: Re: GHC 'hslibs/win32/Win32Spawn.hs' syntax error? | | hslibs/win32 is being kept compilable & working as our partial Win32 | API binding. | | I've never tried using libraries/Win32; its re-introduction of a dependency | on GreenCard + its untested nature is why I've been reluctant to go near it. | | --sigbjorn | | ----- Original Message ----- | From: "Simon Peyton-Jones" <simonpj(a)microsoft.com> | To: "Sven Panne" <Sven.Panne(a)aedion.de>; "Sigbjorn Finne" <sof(a)galois.com> | Cc: "Graham Klyne" <gk(a)ninebynine.org>; <cvs-ghc(a)haskell.org> | Sent: Wednesday, January 28, 2004 02:50 | Subject: RE: GHC 'hslibs/win32/Win32Spawn.hs' syntax error? | | | > Sigbjorn, Sven, Alastair | > | > What *is* our story for hslibs/Win32? | > | > We're trying to switch over to libraries/. | > * Is libraries/Win32 a complete replacement for hslibs/Win32? | > * Is it "alive"? (ie. does it work? does anyone use it) | > | > Sorry for my ignorance. It'd be nice to say "don't use hslibs, just use | > libraries".

2 1

Network.URI module
by Graham Klyne 03 Feb '04

03 Feb '04

At 10:44 03/02/04 +0000, Simon Marlow wrote: > > (1) Network.URI > > > > I've written a new parser, and extended the module interface > > slightly, thus: > >[snip] > >If you'd like to become the maintainer of this module and incorporate >your changes, you're entirely welcome (I'm not using this code actively >at the moment). I'd be happy to do that. What are the ground rules for potentially non-backward-compatible changes? What are the procedures for lodging new releases (CVS?). >You mentioned that there were problems with the existing implementation >- perhaps you could explain further? As far as I'm aware, the regular >expression and the test cases were taken directly from RFC 2396, and the >implementation was correct at the time - did something change? The >current testcases are in testsuite/tests/ghc-regress/lib/net/uri001.hs. I don't think it was a problem with the regular expression per se: (a) the regex in RFC2396 doesn't tell you (reliably) if a URI is or is not valid. What it does do is, assuming a valid URI is presented, is pick apart the various components. (b) I would have stuck with the regex-based implementation here, except that the regex module used is not available on Windows. For me, it was easier to construct a URI parser using Parsec, which doesn't depend on system-dependent modules. (c) there are some small changes in syntax that might affect the regex implementation: reserving '[' and ']' for use in IPv6 literals comes to mind. I haven't checked the details. My parser follows the syntax in the RFC2396bis proposal very closely. As such, it will reject some URIs that the regex implementation would accept. My own test suite includes all the RFC2396 test cases. (The RFC2396 proposal already has extensive review and broad consensus in the URI working group; my Haskell work is providing some implementation feedback.) The problems with behaviour of the current implementation that I did note are covered below... > > I have some concerns about the way URI strings are > > reassembled from the > > component parts using the current URI module interface (e.g. > > problem with > > empty fragment handling noted in a previous message). I > > think the URI > > implementation should be changed so that all the punctuation > > characters > > ("//", "?", "#", etc.) are stored as part of the component > > values in a URI > > structure, but I don't know what impact that might have on > > existing code. > >If that's an unforced change I'd vote to keep the current behaviour, to >avoid breaking code. It's not entirely "unforced"... it has to do with the way a URI is stored internally, and the consequences for reconstructing a URI string from the URI components; e.g. file:///path/name is reconstructed as: file:/path/name http://example.org/path/resource# is reconstructed as: http://example.org/path/resource I have a question [1] outstanding with the URI WG about the validity of the first, and do believe that the second is incorrect (there has been some discussion that the presence of a fragment is significant in some web applications). There is a general presumption in Web circles that a URI should be used in exactly the form given; cf. [2]. [1] http://www.w3.org/mid/5.1.0.14.2.20040202132114.00bd6ec8@127.0.0.1 (Can't get proper URI yet ... lists.w3.org is down as I write) [2] http://www.w3.org/2001/tag/webarch/#lc-uri-chars I suppose it would be possible to make a new implementation of the URI structure that presents the same interface, but remembers the presence of empty fields, but I'm concerned that would be locking in undesirable complexity and propagating a debatable design. (Question: why would one wish the URI components to be stored without their distinguishing punctuation?) ... Here's a proposal: (a) change URI thus: [[ data URI = URI { uriScheme :: String -- ^ @http:@ , uriAuthority :: String -- ^ @//www.haskell.org@ , uriPath :: String -- ^ @/ghc@ , uriQuery :: String -- ^ @?query@ , uriFragment :: String -- ^ @#frag@ } ]] (b) implement access functions that behave like the original field selectors. Then the visible change in behaviour would be that 'show' of any URI would reconstruct exactly the string supplied to construct it. If it turns out that the alternative access functions are not needed, they could be dropped in a later revision (hmmm... do Haskell impleemnetations offer a deprecated flag?). #g ------------ Graham Klyne For email: http://www.ninebynine.org/#Contact

1 0