- Haskell-Cafe - Haskell.org

Fwd: [Haskell] ANNOUNCE: Process library (for dataflow-oriented programming?)
by Bulat Ziganshin 13 Dec '05

13 Dec '05

This is a forwarded message From: Bulat Ziganshin <bulatz(a)HotPOP.com> To: haskell(a)haskell.org Date: Thursday, December 08, 2005, 1:36:05 AM Subject: [Haskell] ANNOUNCE: Process library (for dataflow-oriented programming?) ===8<==============Original message text=============== Hello haskell, Joel's program (discussed in cafe), which now uses MVars instead of Channels to send data between threads, may be a good example of dataflow-driven program: it consists of many hundreds of threads and when one thread sends data to another through MVar, this thread in most cases goes to sleep until receiving thread will process previous value of this MVar. so, threads are waked up and asleep according to passing values between them, and the whole program executes in order defined by these data dependencies, as opposite to the order of program statements one year ago i developed small library, which can be helpful if you want to use such style of programming. its ideas are modelled after Unix pipes, which are widely used to assemble complex data processing "engines" from simple "details". really this library is very thin layer over direct using of forkOS, channels and MVars; nevertheless, is is very convenient and beatiful you can download library as http://freearc.narod.ru/Process.tar.gz this page also contains sources of my program where you can find examples of using library in real toy :) below is a guide to library usage to create pipe, which contains 3 processes - "producer", "transformer" and "consumer": runP ( producer |> transformer |> consumer ) each process in pipe runned in separate Haskell thread. process is represented by ordinary Haskell function which gets an additional parameter - handle, which can be used to receive data from previous process in pipe (using receiveP) and send data to the next process (using sendP). for example, abovementioned processes can be implemented as: producer handle = mapM_ (sendP handle) [1..10] transformer handle = replicateM_ 10 $ do x <- receiveP handle sendP handle (x*2) consumer handle = replicateM_ 10 $ do x <- receiveP handle print x if first process in pipe tries to use receiveP or last process in pipe tries to use sendP, then run-time exception is generated. number of processes in pipe can be arbitrary. because each process is just ordinary Haskell function, you can add additional parameters to processes when constructing pipes: runP ( producer |> multiple 2 |> multiple 3 |> consumer ) multiple n handle = replicateM_ 10 $ do x <- receiveP handle sendP handle (x*n) moreover, you can construct pipe or part of it as ordinary data value, which then can be runned by runP: let pipe = case multipliers of [x] -> multiple x [x,y] -> multiple x |> multiple y [x,y,z] -> multiple x |> multiple y |> multiple z _ -> \handle -> fail "Zero or too much multipliers" runP ( producer |> pipe |> consumer ) there is also "back channel", which can be used to "return" data to previous process in the pipe, its operations is send_backP and receive_backP. it can be used to return acknowledgments, synchronize processes or to return resources back. brief example of its usage: producer: sendP pipe (buf,len) consumer: ; ; (buf,len) <- receiveP pipe ; hPutBuf file buf len ; send_backP pipe () receive_backP pipe ; --now we know that buf is free ; (i organized lines to show execution order) if processes joined in pipe with "|>" then channel between them uses MVar, so at any moment it may contain no more than 1 element. if channel between two processes is created with "|>>>" then Chan is used, which can contain arbitrary number of data items. be careful with such channels, because they can grow to unlimited size. "|>" and "|>>>" can be arbitrarily combined in one pipe: runP ( producer |>>> multiple 2 |> multiple 3 |>>> consumer ) back channel (used by send_backP and receive_backP) are always multi-element (uses Chan) runP returns when all processes in pipe are finished. if any process in pipe generates uncaught exception, then all processes in pipe are killed and this exception is re-raised in thread called runP pipe or single process can also be runned in background using runAsyncP: handle <- runAsyncP (multiple 2) handle returned here can be used to interact with first and last processes in pipe, in contrast to runP: handle <- runAsyncP (multiple 2) sendP handle 1 res <- receiveP handle of course, pipe runned asynchronously is not required to perform input, output, or both: handle <- runAsyncP ( producer |> transformer ) handle <- runAsyncP ( transformer |> consumer ) handle <- runAsyncP ( producer |> transformer |> consumer ) currently channels to "open ends" of background pipe are always one-element (uses MVars). types of channels inside pipe are determined, as usual, by using the "|>" or "|>>>" operator you can wait for finishing of asynchronous pipe with "joinP handle" you can "replace" process, which is runned in pipe, with another process or pipe by using runFuncP: transformer handle = runFuncP ( multiple 2 |> multiple 3 ) (receiveP handle) (send_backP handle) (sendP handle) (receive_backP handle) actually, runFuncP just executes its pipe using 4 supported functions for interaction with first and last pipe processes. to make this obvious i will say how runP can be implemented in terms of runFuncP: runP p = runFuncP p (error "First process in runP tried to receive") (error "First process in runP tried to send_back") (error "Last process in runP tried to send") (error "Last process in runP tried to receive_back") runFuncP returns when all processes in its pipe are finished. using it, you can create unlimited number of scenarios: running several runFuncP sequentially to deal with different parts of your data, process part of data itself and part with runFuncP, "redirect" your input to process runned by runFuncP but consume output from runFuncP in other way: consumer handle = runFuncP ( multiple 4 ) (receiveP handle) undefined print -- output action undefined runFuncP can also be used in other cases when we need to run single process or whole pipe "imitating" its interaction with "external world" by some functions: runFuncP transformer readLn -- input action undefined print -- output action undefined it will also be interesting to have some functions which can just insert "filtering" process or pipe on input or output side of current process: transformer channel = do channel <- insertInputFilterP (multiple 2) channel channel <- insertOutputFilterP (multiple 4) channel x <- receiveP channel sendP channel (x+1) ... but i don't need it and therefore this currently is not implemented the library also don't contains routines to check that input is ready or output can be done (because i don't need it and it is a bad programming style), nor routines to check for EOF (i prefer to encode this explicitly in structure of data sent across channels; you also can wrap your data in Maybe type and use Nothing to encode EOF) -- Best regards, Bulat mailto:bulatz@HotPOP.com _______________________________________________ Haskell mailing list Haskell(a)haskell.org http://www.haskell.org/mailman/listinfo/haskell ===8<===========End of original message text=========== -- Best regards, Bulat mailto:bulatz@HotPOP.com

2 2

Bringing Erlang to Haskell
by Joel Reymont 13 Dec '05

13 Dec '05

Folks, I love the Erlang multi-processing experience and think that a lot of the mistakes that I made could be avoided. What I want to have is 1) Processes, aka threads with single-slot in/out mailboxes 2) A facility to keep a list of such processes and send events to them using their process id 3) A socket reader/writer abstraction that communicates with the outside world using using its mailboxes Probably some other things but I would start with the above. I also want to use STM for this. One particular thing that bugs me is that I cannot really use TChan for thread mailboxes. I don't think I experienced this problem with Erlang but using a TChan with a logger thread quickly overwhelms the logger and fills the TChan and a lot (hundreds? thousands) of other threads are logging to it. Someone said it's because the scheduler would give ther other threads proportionally more attention. I found single-slot mailboxes (TMVar) to work much better as they pace the overall message flow. Using them means that asynchronous messages cannot be implemented, though. Please correct me if I'm wrong. I'll update my blog as I move forward. Thanks, Joel -- http://wagerlabs.com/

5 10

trick to easily generate Eq/Ord instances
by Bulat Ziganshin 13 Dec '05

13 Dec '05

Hello sometimes, Eq/Ord classes can't be derived automatically because we need to comare only part of fields. in such situations i use the following trick to easify generation of class instances: data ArchiveBlock = ArchiveBlock { blArchive :: Archive , blType :: BlockType , blCompressor :: Compressor , blPos :: Integer , blOrigSize :: Integer , blCompSize :: Integer , blCRC :: CRC , blFiles :: Int } instance Eq ArchiveBlock where (==) = map2eq $ map3 (blArchive,blPos,blCRC) instance Ord ArchiveBlock where compare = map2cmp $ map2 (blArchive,blPos) {- instance Ord ArchiveBlock where compare = map2cmp blPos -- for comparision on just one field -} -- Utility functions map2 (f,g) a = (f a, g a) map3 (f,g,h) a = (f a, g a, h a) keyval f x = (f x, x) -- |Return pair containing computed key and original value map2cmp f x y = (f x) `compare` (f y) -- |Converts "key_func" to "compare_func" map2eq f x y = (f x) == (f y) -- |Converts "key_func" to "eq_func" -- Best regards, Bulat mailto:bulatz@HotPOP.com

2 1

The Price of Performance
by Joel Reymont 13 Dec '05

13 Dec '05

I thought this would be of interest to the Haskell community: http://www.acmqueue.com/modules.php?name=Content&pa=showpage&pid=330 "The ever-growing popularity of small multiprocessors is exposing more programmers to parallel hardware. More tools to spot correctness and performance problems are becoming available (e.g., thread checkers8 and performance debuggers9). Also, a few expert programmers can write efficient threaded code that is in turn leveraged by many others. Fast-locking and thread-efficient memory allocation libraries are good examples of programming work that is highly leveraged." Viva le STM! -- http://wagerlabs.com/

1 0

Tricky exception handling
by Joel Reymont 13 Dec '05

13 Dec '05

Folks, I have a thread that launches socket reader and writer threads. My intent is to notify the main thread when an exception is caught by the reader/writer. Is this code proper? --- withException :: (Event a -> IO ()) -> IO () -> IO Bool withException post action = handle (\e -> do case e of AsyncException ThreadKilled -> return () _ -> post $ NetworkError e return False ) $ do { action; return True } writeLoop :: (Event a -> IO ()) -> Handle -> (SSL, BIO, BIO) -> IO () writeLoop post h ssl = do go <- withException post $ do cmd <- read h ssl post $! Cmd $! cmd when go $ writeLoop post h ssl --- Thanks, Joel -- http://wagerlabs.com/

2 4

Opening the same file multiple times
by Einar Karttunen 13 Dec '05

13 Dec '05

Hello It seems that opening the same file multiple times (one writer and multiple readers) is not supported at least on *nix with GHC. I want to use one Handle to use append data till the end of the file while other Handles perform random access IO with seeks on the file. Sharing the same Handle for all the threads is not possible since they perform seeks and may thus mess each other up. Hiding the Handle behind a mutex would limit concurrency more than I like. Thus I wanted to open multiple Handles to the file, but this seems quite hard. My best guess is to create a function like: #ifdef mingw32_HOST_OS openUnlocked fn mode = openBinaryFile fn mode #else openUnlocked fn mode = withMVar mutex $ do h <- openBinaryFile fn mode fd <- handleToFd h unlockFile $ fromIntegral fd return h {-# NOINLINE mutes #-} mutex = unsafePerformIO $ newMVar () #endif Is there really no simpler solution? - Einar Karttunen

3 4

[Newbie] Why or why not haskell ?
by Christophe Plasschaert 12 Dec '05

12 Dec '05

First, i'd like to say good day to everybody. I didn't find enough information to anwser this question: Being interesting in learning another way of programming besides C and perl i started watching other languages. These are my candidates: - ada; - erlang; - clisp or scheme; and of course - haskell. Haskell seems very interesting indeed (monadic, STM, a good library). Maybe some of you know some or all of these languages and can explain me some pros/cons about them especially for network programming. Thanks for your attention. Have a nice day. C. Plasschaert

10 16

Re: [Haskell-cafe] Opening the same file multiple times
by Donn Cave 12 Dec '05

12 Dec '05

Quoth Einar Karttunen <ekarttun(a)cs.helsinki.fi>: | On 11.12 22:26, Donn Cave wrote: | > Quoth Einar Karttunen <ekarttun(a)cs.helsinki.fi>: | > | It seems that opening the same file multiple times (one writer | > | and multiple readers) is not supported at least on *nix with | > | GHC. I want to use one Handle to use append data till the | > | end of the file while other Handles perform random access | > | IO with seeks on the file. | > | > How is it not supported? What happens with something like this ... | *** Exception: z: openFile: resource busy (file is locked) Oh. Well, that's unfortunate, but at least I see it is documented. I guess that explains why your solution uses unlockFile. I suppose you might be able to use Posix.IO functions instead, but that would probably be even more trouble than your work-around. Donn Cave, donn(a)drizzle.com

2 3

RE: [Haskell-cafe] Opening the same file multiple times
by Simon Marlow 12 Dec '05

12 Dec '05

On 12 December 2005 17:53, Donn Cave wrote: > Quoth Einar Karttunen <ekarttun(a)cs.helsinki.fi>: > ... >> *** Exception: z: openFile: resource busy (file is locked) > > Now that I have access to a platform where ghc builds, I can > duplicate your results - and in case it helps, here's another > work-around. Apparently this "feature" uses POSIX filesystem > locks, because it appears that we can exploit the brain-damaged > semantics of that system: when a process closes a file, all > its locks are cleared - whether they were acquired through that > file descriptor or another unrelated open. It may be the first > time anyone has ever benefited from this! > > import IO > > main = do > fc <- openFile "z" ReadMode -- sacrificial handle > fr <- openFile "z" ReadMode > hClose fc > fa <- openFile "z" AppendMode > hPutStr fa "append this line\n" > hGetLine fr >>= print > > If you may need to open more read handles later, you can add > another extra close after the append handle. Thank you (I suppose :-) for exposing a bug in GHC's file locking. It doesn't have anything to do with POSIX filesystem locks, FWIW, it's just a bug, in some very old code. Cheers, Simon

1 0

syscall, sigpause and EINTR on Mac OSX
by Joel Reymont 12 Dec '05

12 Dec '05

Folks, I'm kind of stuck on this and can't figure it out... Take a look at the trace below, this is Mac OSX Tiger. (gdb) where #0 0x90006068 in syscall () #1 0x9004420c in sigpause () #2 0x001791b8 in awaitUserSignals () at Signals.c:256 #3 0x0012e1a8 in schedule (mainThread=0x1300360, initialCapability=0x0) at Schedule.c:518 #4 0x0012f340 in waitThread_ (m=0x1300360, initialCapability=0x0) at Schedule.c:2156 #5 0x0012f224 in scheduleWaitThread (tso=0x15c0000, ret=0x0, initialCapability=0x0) at Schedule.c:2050 #6 0x0012ba18 in rts_evalLazyIO (p=0x215254, ret=0x0) at RtsAPI.c:459 #7 0x0001ea7c in main (argc=4, argv=0x0) at Main.c:104 (gdb) p16 $r22 0x40: Cannot access memory at address 0x40 My program is currently stuck here. The man pages say that sigpause will only terminate by being interrupted and EINTR will be the errno. EINTR is signal 2, the same one that I'm trapping and the one sent when ^C is pressed. This probably explains why my signal handler is getting hit from time to time without any input from me. My program then exits and leaves me very puzzled as the whole thing looks like someone sneaked up to my keyboard and hit ^C to interrupt the program. I'm not making any sense of this. Why would someone be calling sigpause and making it interrupt my program? How can I work around this? I guess I'll poke around the ghc internals a bit and see what's cooking. Thanks, Joel -- http://wagerlabs.com/

6 37