November 2007 - Haskell-Cafe

Guidance on using asynchronous exceptions
by Yang 16 Nov '07

16 Nov '07

To follow up on my previous post ("Asynchronous Exceptions and the RealWorld"), I've decided to put together something more concrete in the hopes of eliciting response. I'm trying to write a library of higher-level concurrency abstractions, in particular for asynchronous systems programming. The principal goal here is composability and safety. Ideally, one can apply combinators on any existing (IO a), not just procedures written for this library. But that seems like a pipe dream at this point. In the code below, the running theme is process orchestration. (I've put TODOs at places where I'm blocked - no pun intended.) I'm currently worried that what I'm trying to do is simply impossible in Concurrent Haskell. I'm bewildered by the design decisions in the asynchronous exceptions paper. I'm also wondering if there are any efforts under way to reform this situation. I found some relevant posts below hinting at this, but I'm not sure what the status is today. (Something like this is straightforward to build if I abandon Concurrent Haskell and use cooperative threading, and if the operations I wanted to perform could be done asynchronously.) Relevant papers --------------- http://citeseer.ist.psu.edu/415348.html http://research.microsoft.com/users/simonpj/papers/concurrent-haskell.ps.gz http://www.haskell.org/~simonmar/papers/web-server.ps.gz Relevant posts/threads ---------------------- http://osdir.com/ml/lang.haskell.prime/2006-04/msg00032.html http://osdir.com/ml/lang.haskell.general/2001-11/msg00131.html http://www.haskell.org/pipermail/haskell-prime/2006-April/001280.html http://www.haskell.org/pipermail/haskell-prime/2006-April/001290.html http://www.nabble.com/throwTo---block-statements-considered-harmful-tf27802… http://www.nabble.com/What-guarantees-(if-any)-do-interruptible-operations-… Misc ---- http://lambda-the-ultimate.org/node/1570 Advanced Exception Handling Mechanisms http://www.springerlink.com/content/3723wg2t81248027/ http://64.233.169.104/search?q=cache:c4pS0FDKMXcJ:www.cs.ioc.ee/tfp-icfp-gp… http://64.233.169.104/search?q=cache:hmC-jl-iNkoJ:www.jot.fm/issues/issue_2… http://www.mathematik.uni-marburg.de/~eden/paper/edenEuropar03.pdf Code ==== module Main where import Control.Concurrent import Control.Concurrent.MVar import Control.Exception import Control.Monad import Prelude hiding (log) import System.IO import System.Posix.Signals import System.Process log = putStrLn startProc cmd = runCommand cmd stopProc p = terminateProcess p waitProc p = waitForProcess p -- Run a process, blocking on it until it exits. If we're interrupted, -- terminate the process. (IIRC, terminateProcess issues SIGTERM, and -- the documentation is buggy; more detailed code should go here later -- to retry with SIGKILL.) runProc cmd = do log "launching proc" p <- startProc cmd waitProc p -- TODO allow interrupts only at this point `finally` ( log "stopping" >> stopProc p >> log "stopped" ) -- Sleep for n seconds. timeout n = do log "sleeping" threadDelay (n * 1000000) -- TODO allow interrupts only at this point log "waking" -- TODO is there any way to block *only* the Cancel exception? (Even -- if this could be done, though, it's still not a modular approach.) spawn :: IO a -> (a -> IO ()) -> IO ThreadId spawn f y = forkIO (block (f >>= y)) -- The any/sum/choice combinator. On return, guarantee that both tasks -- have stopped. (<|>) :: IO a -> IO b -> IO (Either a b) a <|> b = do result <- newEmptyMVar :: IO (MVar (Either a b)) tida <- newEmptyMVar :: IO (MVar ThreadId) tidb <- newEmptyMVar :: IO (MVar ThreadId) let yield lr x = do let name = case lr x of Left _ -> "a" Right _ -> "b" log $ "saving result of " ++ name putMVar result (lr x) log $ "saved result of " ++ name let other = case lr x of Left _ -> tidb Right _ -> tida log "taking other" t <- takeMVar other log "killing other" -- Later: replace the following with a throwTo -- so as to notify (rather than kill) the thread -- with a Cancel killThread t ta <- spawn a (yield Left) tb <- spawn b (yield Right) putMVar tida ta putMVar tidb tb log "waiting for result" res <- takeMVar result -- TODO wait for both tasks to have stopped log "returning result" return res -- simple test -- cmd1 = "for i in `seq 1`; do sleep 1; echo hello; done" cmd2 = "for i in `seq 3`; do sleep 1; echo world; done" main = do -- TODO for some reason, cmd2 doesn't get terminated. result <- runProc cmd1 <|> runProc cmd2 case result of Left _ -> putStrLn "finished process" Right _ -> putStrLn "got exception"

2 1

Performance problems with parallelizing QuickCheck using channels (Control.Concurrent.Chan)
by Gwern Branwen 15 Nov '07

15 Nov '07

So I've been trying to get my QuickCheck tests to run in parallel. I did take a look at Don's Parallel QuickCheck library <http://www.cse.unsw.edu.au/~dons/pqc.html>, but I didn't like how much code it had in it and I figured it'd be a good exercise to try to do myself. After quite a lot of help from the good folk of #haskell, I eventually came up with this: module Pcheck (parTest, parCheck) where import Control.Monad (replicateM_, liftM) import Control.Concurrent.Chan (newChan, writeChan, getChanContents) import Control.Concurrent (forkIO) import Test.QuickCheck (quickCheck', Testable()) -- | Takes a list of functions using parCheck, and returns True iff all return -- True. Evaluates them in parallel. parTest :: [IO Bool] -> IO Bool parTest = andTest . parList where andTest :: IO [Bool] -> IO Bool andTest = liftM and {- | Test in parallel. Forks off a QuickCheck 'n' times; QuickCheck tests using the proposition 't'. Returns True if all tests were passed, else False. Should be run with parallelizing options like with +RTS -N4 -RTS &etc. -} parCheck :: (Testable prop) => prop -> Int -> IO Bool parCheck t n = do chan <- newChan replicateM_ n $ forkIO $ (writeChan chan) =<< (quickCheck' t) liftM (and . take n) $ getChanContents chan -- | Takes a list of functions (presumably using parCheck) and evaluates all in parallel. parList :: [IO a] -> IO [a] parList fs = do chan <- newChan mapM_ (\m -> forkIO $ m >>= writeChan chan) fs liftM (take n) $ getChanContents chan where n = length fs I liked how simple the Channels library <http://haskell.org/ghc/docs/latest/html/libraries/base-3.0.0.0/Control-Conc…> seemed to be; I could just pass the channel as an argument and have every forkIO'd test simply chuck its Boolean result into it when done - which seem'd much simpler than using MVars and the other techniques for returning stuff from forkIO threads. And so it compiles, it runs tests correctly, and so on. But the problem is that it does so slowly. I have another module of equations about nuclear bombs called nuke.hs, which has a number of QuickCheck properties defined. Here's what happens when main is defined as 'parTest [the various tests..]': ./nuke +RTS -N7 -sstderr -RTS 40.57s user 46.55s system 116% cpu 1:14.61 total ./nuke +RTS -N6 -sstderr -RTS 40.72s user 47.66s system 117% cpu 1:15.50 total ./nuke +RTS -N5 -sstderr -RTS 42.33s user 49.08s system 116% cpu 1:18.67 total ./nuke +RTS -N4 -sstderr -RTS 43.71s user 48.41s system 117% cpu 1:18.48 total ./nuke +RTS -N3 -sstderr -RTS 41.51s user 48.25s system 114% cpu 1:18.10 total ./nuke +RTS -N2 -sstderr -RTS 42.28s user 47.18s system 115% cpu 1:17.39 total ./nuke +RTS -N1 -sstderr -RTS 27.87s user 18.40s system 99% cpu 46.498 total (From <http://hpaste.org/3886#a6>; compiled as "=ghc -v --make -threaded -O2 ./nuke.hs".) For some reason, running the parallel tests with a single thread is faster than running with 4 threads (I have a quad-core Intel processor)? I find this counter-intuitive to say the least. the par* functions are indeed operating in parallel, as evidenced by it using more than 100% CPU time, or, running on multiple cores, and all the tests are passed as True in both -N1 and -N[2-7] versions, so -N1 can't be bailing out early due to "and"'s laziness, and in general everything seems to be written correctly. I am perplexed by this. Is Chan simply a very inefficient way of parallelizing things? Is it not as parallel as I think? Or am I missing something else entirely? (Attached is the source of nuke.hs and pcheck.hs, as well as some data from -sstderr.) -- gwern .45 GIGN jya. wire ISI SADCC JPL embassy Recon World

1 0

(general question) Approaches for storing large amount of simple data structures
by bbrown 15 Nov '07

15 Nov '07

I have a project where I want to store a data structure on a file, binary or ascii. And I want to use haskell to read and write the file. I will have about half a million records so it would be nice if the format was able to load quickly. I guess I could, but I kind of want to avoid using XML. I have the following structure in pseudo code. A URL -> id -> keywords associated with that URL -> title associated with that URL -> links contained in that URL. (0 ... N) What is an approach for saving 500 thousand of those types of records and where I can load the data into a haskell data type. -- Berlin Brown [berlin dot brown at gmail dot com] http://botspiritcompany.com/botlist/?

2 1

Asynchronous exceptions and the RealWorld
by Yang 15 Nov '07

15 Nov '07

- I want to set timeouts on various IO operations in an existing library, ftphs. However, even if I wrap every call in `block`, it's unclear whether ftphs has been coded to deal with asynchronous exceptions, or whether state could be rendered inconsistent. Is there any solution for this aside from auditing/changing ftphs? - I can't find any documentation on which calls are (either directly or indirectly, or parametrically) "interruptible operations" (to use the term from the Async Exceptions paper). For instance, I experimentally determined that hPutStrLn on a socket is *not*. Does such documentation exist anywhere? Furthermore, is there any way to embed this information in the type system, so that Haskellers don't produce async-exception-unaware code? (Effectively, introducing checked interrupts?) - The above finding of hPutStrLn (as an example) is somewhat surprising, since the socket buffer may be full, causing the call to block indefinitely. We can unblock that call, but unless there's a way to atomically reinstate the block and save the result of the computation (the unit value), this effectively makes it impossible to determine whether the hPutStrLn returned. (Admittedly, this is a contrived example, since whether a call to hPutStrLn returns is usually not a useful result - but this is the first thing I tried, and I'm guessing there are more such IO operations that actually yield useful results.) - Are there any higher-level concurrency libraries that actually tackle these hand-in-hand issues (safe asynchronous interruptible IO, two-phase-commit to prevent dropping results, choice, etc.)? I tried searching for a while, coming up with PiMonad, HAppS.Util, MissingH.Threads.Timeout, but none of them really address these issues. Perhaps this is just impossible in Concurrent Haskell, and one would need to resort to the sure-fire home-rolled solution using delimited continuations and cooperative scheduling for any serious concurrency abstractions for systems programming. Thanks in advance for any answers!

1 0

Re: Flymake Haskell
by Daisuke IKEGAMI 15 Nov '07

15 Nov '07

Dear Stefan and Haskell-Cafe, Thanks to keeping your interest to the flymake-mode for Haskell. Stefan wrote: > Could you explain to me what flycheck_haskell.pl does, and give an > example of a problematic situation solved by the use of > flycheck_haskell.pl. Sure. The perl script flycheck_haskell.pl, which is presented at EmacsWiki, http://www.emacswiki.org/cgi-bin/emacs/FlymakeHaskell edits the warning and error messages as an one line from the output of ghc. Now I'll describe how to solve the perl script the problem for using the flymake-mode with ghc. Here is an example of an error message by ghc. Foo.hs:4:4: No instance for (Num Char) arising from the literal `0' at Foo.hs:4:4 Possible fix: add an instance declaration for (Num Char) In the expression: 0 In the definition of `i': i = 0 where the Foo.hs has an explicit error as follows: ---- module Foo where i :: Char -- a type error here i = 0 ---- The perl script removes newlines and white spaces from the error message from ghc and then print it as follows (long one line): Foo.hs:4:4: No instance for (Num Char) arising from the literal `0'... The reason why I needed the perl script is that the current flymake-mode assumes that the syntax checker must print warning/error messages as an 'one' line for each and the message must have the following information: - filename (required) - line number (required) - column number (optional) - warning/error message (required) The above requirement is the problem when we meet to use flymake-mode with ghc alone. If you want to know the detail about the original flymake-mode, please read the definition of two functions - 'flymake-parse-err-lines - 'flymake-parse-line instead of asking me more further. I'm a pragmatic programmer who knows a few about EmacsLisp and can't explain the flymake-mode clearly. However, I greatly appreciate any discussion about the flymake-mode for Haskell here. > > Note that it will not work in the future, since the flymake-mode > > has been developed so actively. > Huh? Where? I forgot to write the following sentence in the previous email: the flymake-mode has been developed so actively at not the official site http://flymake.sourceforge.net/ but http://cvs.savannah.gnu.org/viewvc/emacs/emacs/lisp/progmodes/flymake.el The latest changes had done on 2 weeks and 4 days ago. I would like to thank Don Stewart for his kindly comments on IRC and that he introduced Flymake Emacs at the latest Haskell news. Gwern Branwen helps me a lot and add a section about Flymake Haskell to the Haskell wiki. http://haskell.org/haskellwiki/Haskell_mode_for_Emacs I would like to thank also anonymous reviewers on EmacsWiki. Best wishes, ike

3 3

Vertigo / GHC / Memo
by Peter Verswyvelen 15 Nov '07

15 Nov '07

Hi Haskell Lovers, Im trying to compile Vertigo (http://conal.net/Vertigo) which seems to be a module from which I can really learn a lot of stuff. However, it seems to use a Memo module, which does not seem to be part of GHC (anymore?) As far as I understand, it uses a memo function to make sure that when performing symbolic differentiation, it does not get stuck in an endless loop. As I realize no generic memoization function can be made, I guess it got removed at some point? Can anyone give me a hint how to find a suitable Memo module for Vertigo? As a sidenote, I must say the memoization section on Haskell.org is not really helpful for a newbie like me... Thanks, Peter PS: I adapted the following code from SOE, but I'm not sure it does the job correctly (it will certainly be slow...) {-# OPTIONS_GHC -fglasgow-exts #-} module Memo(memo) where import Data.IORef import System.IO.Unsafe memo :: Eq a => (a->b) -> (a->b) memo f = unsafePerformIO $ do cache <- newIORef [] return $ \x -> unsafePerformIO $ do vals <- readIORef cache case x `inCache` vals of Nothing -> do let y = f x modifyIORef cache ((x,y):) return y Just y -> do return y inCache :: Eq a => a -> [(a,b)] -> Maybe b x `inCache` [] = Nothing x `inCache` ((x',y'):xys) = if x == x' then Just y' else x `inCache` xys

1 0

Using Data.Binary for compression
by Chad Scherrer 15 Nov '07

15 Nov '07

Hi, I'd like to be able to use Data.Binary (or similar) for compression. Say I have an abstract type Symbol, and for each value of Symbol I have a representation in terms of some number of bits. For compression to be efficient, commonly-used Symbols should have very short representations, while less common ones can be longer. Since an encoding like [Bool] would be really inefficient for this (at least I think it would, though some fancy fusion tricks might be able to help), I was thinking a reasonable approach might be to use Word8 (for example), and then specify a number of bits n, indicating that only the first n bits are to be written to the compressed representation. I was looking at the internals of Data.Binary, and saw it seems that PutM could be used for this purpose (was something like this its original purpose?). Today, I put this together: ------------------------------------ type BitRep = Word8 type NBits = Int type MyBits = (BitRep, NBits) (#) :: MyBits -> MyBits -> PutM MyBits (a, m) # (b, n) = case (a .|. (b `shiftR` m), m + n) of ans@(ab, s) -> if s < 8 then return ans else putWord8 ab >> return (b `shiftL` (8 - m), s - 8) ------------------------------------ Then, it would be easy enough to map [Symbol] -> [MyBits], and then use something like foldM (#) to get into the PutM monad. A couple of questions: (1) Am I reinventing the wheel? I haven't seen anything like this, but it would be nice to be a bit more certain. (2) This seems like it will work ok, but the feel is not as clean as the current Data.Binary interface. Is there something I'm missing that might make it easier to integrate this? (3) Right now this is just proof of concept, but eventually I'd like to do some performance tuning, and it would be nice to have a representation that's amenable to this. Any thoughts on speeding this up while keeping the interface reasonably clean would be much appreciated. Thanks! -Chad

6 7

cabal Main-Is restriction
by Nicolas Frisby 15 Nov '07

15 Nov '07

It seems the meaning of the -main-is switch for GHC and the Main-Is build option for Cabal executables differ. With GHC, I can point to any function "main" in any module, but in Cabal I must point to a filename with precisely the module name "Main". This is tying my hands with regard to organizing a default executable and exposing some of its functionality as a library. Is there a way to get around this restriction? Concretely, I want to point Cabal's Main-Is to Program/Main.hs which starts with module Program.Main where instead of just module Main where Is this currently possible? I recognize the "add a separate Program-Main.hs file" workaround, but I'll avoid it if I can. Thanks!

2 1

Haskell Job Opportunity
by Tom Hawkins 15 Nov '07

15 Nov '07

Hello, Eaton (eaton.com, Eden Prairie, MN US) is seeking software engineers for design and verification of electro-hydraulic control systems for industrial, automotive, and aerospace applications. Though I am still trying to get Haskell on the official job description, here are a few of the potential Haskell applications: - Domain specific languages. - Compiler design and embedded code generation. - Model checking and equivalence checking. - SAT decision procedures. - Constrained random simulation. - Software timing analysis. General knowledge of the following would be helpful: - Control theory. - Real-time, embedded programming. - Automotive and industrial systems. - Hydraulics and fluid power. If interested, send me a resume. Thanks! -Tom Hawkins

1 0

Weird ghci behaviour?
by Dan Piponi 15 Nov '07

15 Nov '07

On Unix-like OSes: If I run "ghc test.hs" and then run "ghci test.hs", ghci fails to load up my code. I have to touch test.hs and then run ghci. I can understand ghci refusing to recompile something it thinks it has already compiled. But it appears to refuse to load it into an interactive session - which is less useful. In fact, removing test.hi makes ghci work again. This is ghc 6.6. Anyone else seeing this? -- Dan

18 29