June 2004 - Libraries

Subprocesses (was: [ANNOUNCE] shell-haskell 1.0.0)
by Simon Marlow 08 Jul '04

08 Jul '04

Libraries people, There are currently two ways to create and communicate with external processes: - System.Cmd.system (very basic, doesn't give you access to the process's output but you can use shell primitives to redirect its input/output). - Posix.runProcess (can specify Handles for the process's input/output, but it doesn't create pipes so you can't actually communicate with the subprocess using these Handles.). - POpen.popen (can specify an input String, and gives you lazy output Strings. No interactive communication is possible.) Ok, *three* ways. System.Cmd.system, Posix.runProcess, POpen.popen, and shell-haskell. Er. *Four* ways. David Sankel's shell-haskell creates pipes for stdin/stdout/stderr, and lets you communicate interactively with the sub-process via Handles. It could do with a way to specify the environment and working directory of the sub-process (ala POpen and runProcess), but apart from that it seems to be the most flexible of the four. It could be simplified slightly by using one Handle rather than two for stdin/stdout. We should think about what functionality we want to provide in the core libraries. My preference would be to go for a full version with three pipes (like shell-haskell) and perhaps one or two more specialised versions covering common cases (I don't know what these should be, though). What do people think? Cheers, Simon -----Original Message----- From: haskell-admin(a)haskell.org [mailto:haskell-admin@haskell.org] On Behalf Of David Sankel Sent: 06 May 2003 03:06 To: haskell(a)haskell.org Subject: [ANNOUNCE] shell-haskell 1.0.0 =================== shell-haskell 1.0.0 =================== shell-haskell provides utilities to communicate with other process's via Haskell code. This library can be used for something as simple as getting the output of another program or as complex as interacting with an interpreter. see http://www.electronconsulting.com/shell-haskell for details. David J. Sankel _______________________________________________ Haskell mailing list Haskell(a)haskell.org http://www.haskell.org/mailman/listinfo/haskell

5 10

DData
by Christian Maeder 03 Jul '04

03 Jul '04

Hi, may I warm up an old subject? I would appreciate if at least the Map and the Set module could be added to the base package, soon! DData.Set and the current Data.Set can be merged together (as Sven Panne mentioned earlier). Data.FiniteMap and Data.Map should co-exist for a while. The current Data.FiniteMap and Data.Set have various missing functions and instances. It would make sense to add Data.Bag also (but I don't use Bags currently) (I also don't care about Seq, Queue and Int-specializations.) The last proposal was made by JP Bernardy: http://users.skynet.be/jyp/DData/doc Daan wanted to look into it. My experiments (with large maps) wrt. performance (and memory consumption with and without optimization) of both Map and FiniteMap implementations revealed differences, but no crucial ones. "FiniteMap Int a" performs better than "Map Int a", but not as good as "IntMap a", but the differences are unimportant for us. (We would need a pure HashMap with O(1) lookup while extending the Map) Christian

4 3

binding for libpcap
by Gregory Wright 01 Jul '04

01 Jul '04

Hi, Attached is a binding for libpcap, the packet capture library. I'm looking for comments on how it might be improved, or any particular infelicities that you notice. As I don't use hugs regularly anymore, perhaps someone can point me to a good example of a network library binding that works on both ghc and hugs. It' s tested on FreeBSD (5.2.1-current) but should work with any *nix that can build libpcap. It does not work with Winpcap, a similar library for windows, although it shouldn't be too hard to extend it to do so. The marshalling of packet data is purposely very lightweight (data is returned as Ptr Word8). Having written a lot of embedded software for network test equipment, I prefer the flexibility of being able to marshal into IO-mutable arrays if necessary. But I don't want to impose that on someone who would be satisfied by [Word8]. Best Wishes, Greg Gregory Wright Antiope Associate LLC gwright(a)antiope.com

1 0

Re: [Haskell] Reading a directory tree
by Isaac Jones 29 Jun '04

29 Jun '04

[Followups to libraries(a)haskell.org] This thread makes me want to mention that I'm starting to put together a little library of useful path-based functions (I started it based on OS.Path in Python). Not sure how to organize it, exactly, since some things that might make sense here are already in System.Directory (and some things in there don't make sense in there) and some things may require Posix, and so should be split off into a separate module. My basic requirement is that they should be functions useful for system administration. We've sucked in some functions in from GHC and from Cabal[1]. It would be nice if we could just have a dumping ground for related functions that people have written. For now, if you have darcs[2], you can say: darcs get http://www.syntaxpolice.org/darcs_repos/OS.Path/ (And if you don't have darcs, you can browse around inside there with a web browser to get an idea of how it's shaping up.) That'll download the tree. You can make changes and additions, then use "darcs send" to email the changes to me. I think we should just dump together a bunch of these functions and worry about organizing it later (wiki-style). Here's a function[3] I wrote that's similar to what's mentioned here. It builds a tree out of a file directory. I'd like a function like "walk"[4] that walks this tree and executes an IO action (like printing out all the .txt files or something). It doesn't have to be based on Tree. peace, isaac ------------------------------------------------------------ [1] http://www.haskell.org/cabal [2] http://abridgegame.org/darcs/ [3] -- |Create a tree out of the given starting point. If the starting -- point is a directory, we recurse down and give the entire -- sub-structure, otherwise, we return a single node. -- I think it only requires Posix to test whether this is a directory, -- maybe we can do this with System.Directory? makeDirectoryTree :: FilePath -- ^Starting point (file or directory) -> IO (Tree (FilePath, Bool)) makeDirectoryTree path' = makeDirTree' Nothing path' where makeDirTree' :: Maybe FilePath -- ^Parent Dir -> FilePath -- ^Starting point -> IO (Tree (FilePath, Bool)) makeDirTree' parentIn pathIn = do let fullPath = case parentIn of Just pi' -> (dropLast pi' pathSeparator) ++ [pathSeparator] ++ pathIn Nothing -> pathIn isDir <- pathIsDirectory fullPath contents <- if isDir then getDirectoryContents fullPath else return [] subForest <- mapM (makeDirTree' (Just fullPath)) [c | c <- contents, c /= ".", c /= ['.', pathSeparator] , c /= ['.', '.', pathSeparator], c /= ".."] return $ Node (pathIn, isDir) subForest -- FIX: probably better not to use "error" here, but rather let exception occur. pathIsDirectory :: FilePath -> IO Bool pathIsDirectory p = do existsP <- doesFileExist p existsP2 <- doesDirectoryExist p when (not (existsP || existsP2)) (error $ "File does not exist: " ++ show p) status <- getFileStatus p return $ isDirectory status dropLast [] _ = [] dropLast (h:[]) toDrop | h == toDrop = [] | otherwise = [h] dropLast (h:t) toDrop = h:(dropLast t toDrop) [4] -- |Apply the given function in each directory starting at the -- root. use 'makeDirectoryTree'? FIX: how do we handle symlinks? walk :: FilePath -- ^root -> (FilePath -- current directory -> [FilePath] -- list of files in the directory -> IO a) -- ^visit function (current dir, list of files) -> IO (Tree a)

3 2

HaXml, further updates
by Graham Klyne 29 Jun '04

29 Jun '04

I've done some cleaning up on the code (removing all the old parameter entity handling logic in the parser), and fixed a serious known bug in the general entity substitution filter (recursive GE definitions would blow the stack). The GE substitution logic has been reorganized to separate the various phases. Some new test cases have been added to the test suite. The current code is at: http://www.ninebynine.org/Software/HaskellUtils/20040628-Haxml-1.12.zip http://www.ninebynine.org/Software/HaskellUtils/HaXml-1.12/ There's plenty more that I could do (e.g. see the TODO.txt), but I plan to turn my attention to other areas for the foreseeable future. #g ------------ Graham Klyne For email: http://www.ninebynine.org/#Contact

1 0

DNS and E-Mail library
by Peter Simons 26 Jun '04

26 Jun '04

Hi, I have packaged up two libraries of mine in the hope that someone will find them useful. Here they are: HsDNS: http://cryp.to/hsdns/ An asynchronous DNS resolver based on the GNU Adns library. HsEMail: http://cryp.to/hsemail/ A collection of Parsec combinators, which implement the grammars defined in RFC2821 and RFC2822 (SMTP and Internet Messag Format). The web pages and release archives now contain reference documentation courtesy of Haddock. Peter P. S.: Who would I need to contact to get these links up an the haskell.org/libraries/ page?

1 0

RE: [Haskell-cafe] new Library Infrastructure spec.
by Simon Marlow 24 Jun '04

24 Jun '04

On 24 June 2004 13:18, Henning Thielemann wrote: > 4.1 Package description: > "The exact syntax is still undecided." > > What's about using real Haskell code as package description? The > package file could contain a function that builds a certain data > structure. This would save you from designing a new syntax and would > prevent the user from learning a new description language. One reason for putting the package description into a separate file and specifying that it is pure data rather than a program, is so that external tools can read and modify it. For example, your programming environment might present the package description as a dialog box into which you enter the details (this isn't pie-in-the-sky: we plan to do exactly this in Visual Studio, and there's already a prototype). Cheers, Simon

1 0

new Library Infrastructure spec.
by Isaac Jones 24 Jun '04

24 Jun '04

(followups to libraries(a)haskell.org) Hail Haskellers! A commonly-identified obstacle to the spread of Haskell libraries is that there is no standard way a) for authors to package a library or other tool for distribution to others b) for customers to install such a package Platforms vary. Haskell compilers vary. Packages have dependencies. Etc. Isaac Jones has coordinated an effort to address this problem. We've had a lot of discussion between him and (at least some of) the folk involved in the GHC, Hugs, and nhc implementations. This discussion has led to a new concrete proposed specification[1]. Here it is: http://www.haskell.org/libraryInfrastructure/proposal See also the project home page: http://www.haskell.org/libraryInfrastructure/ We would welcome your feedback, either as a potential library or tool author, or as a potential consumer, or both. The specification isn't complete in every detail, but it seems better to post it before investing in details that may be rendered void by subsequent discussion. We hope this will be an important spec, and will be with us for a long time. So now's the time to look at it. Send feedback to libraries(a)haskell.org. (You can subscribe to this list at: http://www.haskell.org/mailman/listinfo/libraries .) Sincerely, Isaac Jones, Simon Marlow, Ross Paterson, Malcolm Wallace, Simon Peyton Jones [1] This specification differs from the previous proposal both in technical details, and in that it is the combined efforts of the undersigned. Those familiar with the previous proposal will not find large high-level differences, but some details have been made more concrete and some details have changed.

6 11

RE: System.Process
by Simon Marlow 24 Jun '04

24 Jun '04

On 20 June 2004 10:57, David Roundy wrote: >> While doing the Windows port I realised that parts of the interface >> didn't work out too nicely, so I have left out createPipe and >> commandToProcess from the interface. If you think it is imperative >> to have either of these, please yell. > > I don't think either is imperative, but noticed that the documentation > refers to createPipe. Oops, that was left over from a previous version. Now fixed. >> Comments please... perhaps there ought to be some simpler versions of >> the general functions in this library, to cover the more common >> cases? > > While I'm not a fan of the Maybe method of providing optional > arguments, it's hard to see any small set of simpler versions that > one could devise. Having five runProcessX functions doesn't simplify > things. It's probably simplest to let the users define simpler > versions... unless you perhaps wanted to define a runProcess analog > of "system", which differs only in that it accepts a FilePath and > [String] instead of a shell command. Yes, I had in mind providing 'system' and 'rawSystem' replacements too. >> If there are no serious problems, this will be imported into CVS for >> GHC >> 6.4, and possibly future Hugs/nhc98 releases, if there aren't any >> portability problems. > > Sounds great to me, although I should warn you that I haven't actually > tried any of the code... > > Just to make sure I understand things... could runProcess be > implemented in terms of runInteractive as below? Not that this would > be wise--I imagine it goes the other way around--but I want to be > sure that I understand how I'd want to modify things, for example if > I wanted to interactively write to a process, but have its output > logged straight to a file. > > runProcess c as menv (Just inh) (Just outh) (Just errh) = > do (iinh, iouth, ierrh, ph) <- runInteractiveProcess c as menv > instr <- hGetContents inh > outstr <- hGetContents iouth > errstr <- hGetContents ierrh > forkIO $ do hPutStr outh outstr > hClose outh > forkIO $ do hPutStr errh errstr > hClose errh > forkIO $ do hPutStr iinh instr > hClose iinh > return ph Hmm, looks like that might work, yes. It's not quite the same, though: in your version, the inh Handle ends up semi-closed, whereas in the current runProcess implementation it is closed immediately. In your implementation, the outh and errh handles are closed when all the output has been written to them; in my implementaiton they are closed immediately. (I just noticed that my Windows implementation differs from the Unix implementation in this respect). Closing these handles immediately seems more deterministic, but I'm not sure which is the most "useful" behaviour. Cheers, Simon

2 1

HaXml, namespace processing
by Graham Klyne 23 Jun '04

23 Jun '04

Since my last HaXml missive [1], I've managed to get some XML namespace handling code working with HaXml. The namespace processing itself is implemented as a transformation/filter that is applied to XML content after general entity substitution has been performed. The XML content model has been extended to use QNames for element and attribute names (this change was made some time ago), and also to carry XML-Infoset-like information with each element (including a list of in-scope namespaces). When namespace processing is performed, the xmlns attributes are scanned and removed from the XML content. Element and attribute names containing ':' are reprocessed to be in a namespace+local name form. Also, is a default namnespace is in scope, element names not containing a ':' are similarly processed (but not attribute names, because the meaning of unprefixed attributes is determined by the containing XML element). I've created a small suite of test cases for namespace processing, which work by extracting the list of element and attribute QNames from a processed document, and comparing the result with a given list. Also since my last message [1], I've created a test suite for the XML validator, based on the W3C XML conformance data. HaXml passed all but two of these tests without further modification. I've created another snapshot [2] of my work. This represents all of the major functional changes I plan to make at this time, though there is a good deal of code tidying, and a few small features I'd like to add, which I may get round to at some time. My changes are also rather devoid of external documentation. The test harness [3] is one place to look. The namespace handling functions are in module Namespace.hs [4]. #g -- [1] http://www.haskell.org//pipermail/libraries/2004-June/002266.html [2] http://www.ninebynine.org/Software/HaskellUtils/20040622-Haxml-1.12.zip [3] http://www.ninebynine.org/Software/HaskellUtils/HaXml-1.12/test/TestXml.hs [4] http://www.ninebynine.org/Software/HaskellUtils/HaXml-1.12/src/Text/XML/HaX… ------------ Graham Klyne For email: http://www.ninebynine.org/#Contact

1 0