December 2005 - Haskell-Cafe

RE: [Haskell-cafe] Top-level TVars
by Simon Marlow 14 Dec '05

14 Dec '05

On 13 December 2005 18:34, Tomasz Zielonka wrote: > On Tue, Dec 13, 2005 at 06:08:23PM +0000, Joel Reymont wrote: >> Can this be done now or is this a GHC 6.5 feature? >> >> My combination of unsafePerformIO with atomically $ newTVar does not >> seem to be working. > > Here is an example how you can initialize a top-level STM variable. > http://www.uncurry.com/repos/TimeVar/TimeVar.hs > It just forks a new thread inside unsafePerformIO, it runs > "atomically" in it and passes the result through ordinary MVar. A cheaper way is just to seq the top-level TVar at the beginning of Main.main. Cheers, Simon

2 1

RE: [Haskell-cafe] Top-level TVars
by Simon Peyton-Jones 14 Dec '05

14 Dec '05

newTVarIO in the HEAD, and therefore it's in any nightly-build snapshot, which you can freely download. The next major release will be 6.6, but it's a few months off. Meanwhile I hope you can use the workaround that Tomasz posted. Simon | -----Original Message----- | From: haskell-cafe-bounces(a)haskell.org [mailto:haskell-cafe-bounces@haskell.org] On Behalf Of Joel | Reymont | Sent: 13 December 2005 18:08 | To: Haskell-Cafe Cafe | Subject: [Haskell-cafe] Top-level TVars | | Can this be done now or is this a GHC 6.5 feature? | | My combination of unsafePerformIO with atomically $ newTVar does not | seem to be working. | | Thanks, Joel | | P.S. What is the ETA for 6.5? | | On Mon, Dec 05, 2005 at 10:50:13AM -0000, Simon Peyton-Jones wrote: | > | > It turns out to be easy to provide | > | > newTVarIO :: a -> IO (TVar a) | > | > which you can call from inside 'unsafePerformIO'. That means you can | > allocate top-level TVars without fuss. | > | | -- | http://wagerlabs.com/ | | | | | | _______________________________________________ | Haskell-Cafe mailing list | Haskell-Cafe(a)haskell.org | http://www.haskell.org/mailman/listinfo/haskell-cafe

1 0

Top-level TVars
by Joel Reymont 13 Dec '05

13 Dec '05

Can this be done now or is this a GHC 6.5 feature? My combination of unsafePerformIO with atomically $ newTVar does not seem to be working. Thanks, Joel P.S. What is the ETA for 6.5? On Mon, Dec 05, 2005 at 10:50:13AM -0000, Simon Peyton-Jones wrote: > > It turns out to be easy to provide > > newTVarIO :: a -> IO (TVar a) > > which you can call from inside 'unsafePerformIO'. That means you can > allocate top-level TVars without fuss. > -- http://wagerlabs.com/

2 1

FreeBSD: Max # of sockets opened
by Joel Reymont 13 Dec '05

13 Dec '05

Folks, I need some help from those of you with a FreeBSD box. It looks like 'ulimit -n' on FreeBSD lets you have 10k+ file descriptors open per process. FD_SETSIZE is 1024 in the system headers, though. GHC relies on this value (see ghc/rts/Select.c). Normally, you will get the EMFILE error if you try to open more sockets than what is allowed with 'ulimit -n'. If you allow yourself more than 1024 descriptors per process then you do not get this error but... This seems to lead to a situation where you open more than 1024 sockets and shortly afterwards get 'connection resets' for some or all of your sockets. Maybe just those above 1024, I have not determined this precisely. My question is this: is it possible to get a higher number of open sockets by editing the system header files on FreeBSD and recompiling GHC? Has anyone tried this before? How high can you go? Thanks, Joel -- http://wagerlabs.com/

2 3

Fwd: [Haskell] ANNOUNCE: Process library (for dataflow-oriented programming?)
by Bulat Ziganshin 13 Dec '05

13 Dec '05

This is a forwarded message From: Bulat Ziganshin <bulatz(a)HotPOP.com> To: haskell(a)haskell.org Date: Thursday, December 08, 2005, 1:36:05 AM Subject: [Haskell] ANNOUNCE: Process library (for dataflow-oriented programming?) ===8<==============Original message text=============== Hello haskell, Joel's program (discussed in cafe), which now uses MVars instead of Channels to send data between threads, may be a good example of dataflow-driven program: it consists of many hundreds of threads and when one thread sends data to another through MVar, this thread in most cases goes to sleep until receiving thread will process previous value of this MVar. so, threads are waked up and asleep according to passing values between them, and the whole program executes in order defined by these data dependencies, as opposite to the order of program statements one year ago i developed small library, which can be helpful if you want to use such style of programming. its ideas are modelled after Unix pipes, which are widely used to assemble complex data processing "engines" from simple "details". really this library is very thin layer over direct using of forkOS, channels and MVars; nevertheless, is is very convenient and beatiful you can download library as http://freearc.narod.ru/Process.tar.gz this page also contains sources of my program where you can find examples of using library in real toy :) below is a guide to library usage to create pipe, which contains 3 processes - "producer", "transformer" and "consumer": runP ( producer |> transformer |> consumer ) each process in pipe runned in separate Haskell thread. process is represented by ordinary Haskell function which gets an additional parameter - handle, which can be used to receive data from previous process in pipe (using receiveP) and send data to the next process (using sendP). for example, abovementioned processes can be implemented as: producer handle = mapM_ (sendP handle) [1..10] transformer handle = replicateM_ 10 $ do x <- receiveP handle sendP handle (x*2) consumer handle = replicateM_ 10 $ do x <- receiveP handle print x if first process in pipe tries to use receiveP or last process in pipe tries to use sendP, then run-time exception is generated. number of processes in pipe can be arbitrary. because each process is just ordinary Haskell function, you can add additional parameters to processes when constructing pipes: runP ( producer |> multiple 2 |> multiple 3 |> consumer ) multiple n handle = replicateM_ 10 $ do x <- receiveP handle sendP handle (x*n) moreover, you can construct pipe or part of it as ordinary data value, which then can be runned by runP: let pipe = case multipliers of [x] -> multiple x [x,y] -> multiple x |> multiple y [x,y,z] -> multiple x |> multiple y |> multiple z _ -> \handle -> fail "Zero or too much multipliers" runP ( producer |> pipe |> consumer ) there is also "back channel", which can be used to "return" data to previous process in the pipe, its operations is send_backP and receive_backP. it can be used to return acknowledgments, synchronize processes or to return resources back. brief example of its usage: producer: sendP pipe (buf,len) consumer: ; ; (buf,len) <- receiveP pipe ; hPutBuf file buf len ; send_backP pipe () receive_backP pipe ; --now we know that buf is free ; (i organized lines to show execution order) if processes joined in pipe with "|>" then channel between them uses MVar, so at any moment it may contain no more than 1 element. if channel between two processes is created with "|>>>" then Chan is used, which can contain arbitrary number of data items. be careful with such channels, because they can grow to unlimited size. "|>" and "|>>>" can be arbitrarily combined in one pipe: runP ( producer |>>> multiple 2 |> multiple 3 |>>> consumer ) back channel (used by send_backP and receive_backP) are always multi-element (uses Chan) runP returns when all processes in pipe are finished. if any process in pipe generates uncaught exception, then all processes in pipe are killed and this exception is re-raised in thread called runP pipe or single process can also be runned in background using runAsyncP: handle <- runAsyncP (multiple 2) handle returned here can be used to interact with first and last processes in pipe, in contrast to runP: handle <- runAsyncP (multiple 2) sendP handle 1 res <- receiveP handle of course, pipe runned asynchronously is not required to perform input, output, or both: handle <- runAsyncP ( producer |> transformer ) handle <- runAsyncP ( transformer |> consumer ) handle <- runAsyncP ( producer |> transformer |> consumer ) currently channels to "open ends" of background pipe are always one-element (uses MVars). types of channels inside pipe are determined, as usual, by using the "|>" or "|>>>" operator you can wait for finishing of asynchronous pipe with "joinP handle" you can "replace" process, which is runned in pipe, with another process or pipe by using runFuncP: transformer handle = runFuncP ( multiple 2 |> multiple 3 ) (receiveP handle) (send_backP handle) (sendP handle) (receive_backP handle) actually, runFuncP just executes its pipe using 4 supported functions for interaction with first and last pipe processes. to make this obvious i will say how runP can be implemented in terms of runFuncP: runP p = runFuncP p (error "First process in runP tried to receive") (error "First process in runP tried to send_back") (error "Last process in runP tried to send") (error "Last process in runP tried to receive_back") runFuncP returns when all processes in its pipe are finished. using it, you can create unlimited number of scenarios: running several runFuncP sequentially to deal with different parts of your data, process part of data itself and part with runFuncP, "redirect" your input to process runned by runFuncP but consume output from runFuncP in other way: consumer handle = runFuncP ( multiple 4 ) (receiveP handle) undefined print -- output action undefined runFuncP can also be used in other cases when we need to run single process or whole pipe "imitating" its interaction with "external world" by some functions: runFuncP transformer readLn -- input action undefined print -- output action undefined it will also be interesting to have some functions which can just insert "filtering" process or pipe on input or output side of current process: transformer channel = do channel <- insertInputFilterP (multiple 2) channel channel <- insertOutputFilterP (multiple 4) channel x <- receiveP channel sendP channel (x+1) ... but i don't need it and therefore this currently is not implemented the library also don't contains routines to check that input is ready or output can be done (because i don't need it and it is a bad programming style), nor routines to check for EOF (i prefer to encode this explicitly in structure of data sent across channels; you also can wrap your data in Maybe type and use Nothing to encode EOF) -- Best regards, Bulat mailto:bulatz@HotPOP.com _______________________________________________ Haskell mailing list Haskell(a)haskell.org http://www.haskell.org/mailman/listinfo/haskell ===8<===========End of original message text=========== -- Best regards, Bulat mailto:bulatz@HotPOP.com

2 2

Bringing Erlang to Haskell
by Joel Reymont 13 Dec '05

13 Dec '05

Folks, I love the Erlang multi-processing experience and think that a lot of the mistakes that I made could be avoided. What I want to have is 1) Processes, aka threads with single-slot in/out mailboxes 2) A facility to keep a list of such processes and send events to them using their process id 3) A socket reader/writer abstraction that communicates with the outside world using using its mailboxes Probably some other things but I would start with the above. I also want to use STM for this. One particular thing that bugs me is that I cannot really use TChan for thread mailboxes. I don't think I experienced this problem with Erlang but using a TChan with a logger thread quickly overwhelms the logger and fills the TChan and a lot (hundreds? thousands) of other threads are logging to it. Someone said it's because the scheduler would give ther other threads proportionally more attention. I found single-slot mailboxes (TMVar) to work much better as they pace the overall message flow. Using them means that asynchronous messages cannot be implemented, though. Please correct me if I'm wrong. I'll update my blog as I move forward. Thanks, Joel -- http://wagerlabs.com/

5 10

trick to easily generate Eq/Ord instances
by Bulat Ziganshin 13 Dec '05

13 Dec '05

Hello sometimes, Eq/Ord classes can't be derived automatically because we need to comare only part of fields. in such situations i use the following trick to easify generation of class instances: data ArchiveBlock = ArchiveBlock { blArchive :: Archive , blType :: BlockType , blCompressor :: Compressor , blPos :: Integer , blOrigSize :: Integer , blCompSize :: Integer , blCRC :: CRC , blFiles :: Int } instance Eq ArchiveBlock where (==) = map2eq $ map3 (blArchive,blPos,blCRC) instance Ord ArchiveBlock where compare = map2cmp $ map2 (blArchive,blPos) {- instance Ord ArchiveBlock where compare = map2cmp blPos -- for comparision on just one field -} -- Utility functions map2 (f,g) a = (f a, g a) map3 (f,g,h) a = (f a, g a, h a) keyval f x = (f x, x) -- |Return pair containing computed key and original value map2cmp f x y = (f x) `compare` (f y) -- |Converts "key_func" to "compare_func" map2eq f x y = (f x) == (f y) -- |Converts "key_func" to "eq_func" -- Best regards, Bulat mailto:bulatz@HotPOP.com

2 1

The Price of Performance
by Joel Reymont 13 Dec '05

13 Dec '05

I thought this would be of interest to the Haskell community: http://www.acmqueue.com/modules.php?name=Content&pa=showpage&pid=330 "The ever-growing popularity of small multiprocessors is exposing more programmers to parallel hardware. More tools to spot correctness and performance problems are becoming available (e.g., thread checkers8 and performance debuggers9). Also, a few expert programmers can write efficient threaded code that is in turn leveraged by many others. Fast-locking and thread-efficient memory allocation libraries are good examples of programming work that is highly leveraged." Viva le STM! -- http://wagerlabs.com/

1 0

Tricky exception handling
by Joel Reymont 13 Dec '05

13 Dec '05

Folks, I have a thread that launches socket reader and writer threads. My intent is to notify the main thread when an exception is caught by the reader/writer. Is this code proper? --- withException :: (Event a -> IO ()) -> IO () -> IO Bool withException post action = handle (\e -> do case e of AsyncException ThreadKilled -> return () _ -> post $ NetworkError e return False ) $ do { action; return True } writeLoop :: (Event a -> IO ()) -> Handle -> (SSL, BIO, BIO) -> IO () writeLoop post h ssl = do go <- withException post $ do cmd <- read h ssl post $! Cmd $! cmd when go $ writeLoop post h ssl --- Thanks, Joel -- http://wagerlabs.com/

2 4

Opening the same file multiple times
by Einar Karttunen 13 Dec '05

13 Dec '05

Hello It seems that opening the same file multiple times (one writer and multiple readers) is not supported at least on *nix with GHC. I want to use one Handle to use append data till the end of the file while other Handles perform random access IO with seeks on the file. Sharing the same Handle for all the threads is not possible since they perform seeks and may thus mess each other up. Hiding the Handle behind a mutex would limit concurrency more than I like. Thus I wanted to open multiple Handles to the file, but this seems quite hard. My best guess is to create a function like: #ifdef mingw32_HOST_OS openUnlocked fn mode = openBinaryFile fn mode #else openUnlocked fn mode = withMVar mutex $ do h <- openBinaryFile fn mode fd <- handleToFd h unlockFile $ fromIntegral fd return h {-# NOINLINE mutes #-} mutex = unsafePerformIO $ newMVar () #endif Is there really no simpler solution? - Einar Karttunen

3 4