November 2005 - Haskell-Cafe

RE: [Haskell-cafe] Project postmortem
by Simon Marlow 18 Nov '05

18 Nov '05

On 18 November 2005 14:42, Joel Reymont wrote: > On Nov 18, 2005, at 1:55 PM, Simon Marlow wrote: > >> You can get debugging output by compiling your program with -debug, >> and then running it with some of the -D<something> options (use +RTS >> -? for a list, +RTS -Ds is a good one to start with). > > I'm still working on a repro case but here's what I get... > > +RTS -Ds > ... > scheduler: checking for threads blocked on I/O > sched: -->> running thread 1103 ThreadRunGHC ... > sched: --<< thread 1103 (ThreadRunGHC) stopped: is blocked on an MVar > all threads: > thread 1225 @ 0x1539000 is not blocked > thread 1224 @ 0x1506aa4 is not blocked > thread 1223 @ 0x15066a4 is not blocked > ... > scheduler: checking for threads blocked on I/O > sched: -->> running thread 1107 ThreadRunGHC ... > Segmentation fault > > 1107 is not blocked in the list of all threads. What options should I > try next? That doesn't tell us much unfortunately. Can you send a disassembly of the block in which the crash happened? Is it always the same block, BTW? Does changing the heap size (+RTS -H<size>) have any effect? Cheers, Simon

3 3

Re: [Haskell] Re: Making Haskell more open
by Malcolm Wallace 18 Nov '05

18 Nov '05

Ben Moseley <ben_moseley(a)mac.com> writes: > Simon Peyton-Jones <simonpj <at> microsoft.com> writes: > > > ... And avoid > > getting screwed up by malicious folk? > > ... but I believe there are a number of people who regularly > review all the "Recent Changes" and undertake to 'undo' any > malicious/inaccurate modifications. I suspect this would end up adding /more/ work to central maintainers, rather than less. In any case, there is already a Haskell wiki. If it becomes sufficiently rich as a source of information, it will naturally start to take over from the "fixed" webpages. This improved-documentation experiment has already begun some time ago... Regards, Malcolm

9 13

Project postmortem
by Joel Reymont 17 Nov '05

17 Nov '05

Folks, I have done a lot of experiments over the past few weeks and came to a few interesting conclusions. First some background, then issues, solutions and conclusions. I wrote a test harness for a poker server that understands the different binary packets and can send and receive them. The harness launches each "script" in a separate unbound thread that connects to the server via TCP and does its work. The main goals of the project were: easy scripting, very high number of connections from the harness (a few thousand) and running on Windows. I develop on Mac OSX but have a Windows machine for testing and to run the poker server. Another key goal was to support the server encryption. SSL encryption is done in a wierd way that requires attaching read/write OpenSSL BIOs to the SSL descriptor so that SSL encrypts to/from memory. Encrypted chunks are then taken from the BIOs and sent as payload in servver packets. Overall, I probably spent about 4 weeks writing the server and about 2 more weeks grappling with the various issues. The issues centered around 1) the program trashing memory like no tomorrow, 2) intermittent crashes on Windows and 3) not being able to launch a high number of connections on Windows before crashing. I significantly improved trashing of memory by switching to plain Haskell structures from nested lists of wxHaskell-style properties (attr := value). Intermittent crashes were harder to troubleshoot, specially given that things were running smoothly on Mac OSX. Stack traces pointed into libcrypto (part of OpenSSL) and thus to the BIOs that I was allocating. I guesses that OpenSSL was maxing out some resources and closed the leak by explicitly freeing the SSL descriptor which freed the associated BIO structures. Then things got wierder as my program started crashing in a different place entirely with stack traces like this: Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_INVALID_ADDRESS at address: 0x3139322e 0x0027c174 in s8j1_info () (gdb) where #0 0x0027c174 in s8j1_info () #1 0x0021c9f4 in StgRunIsImplementedInAssembler () at StgCRun.c:576 #2 0x0021cdc4 in schedule (mainThread=0x1100360, initialCapability=0x308548) at Schedule.c:932 #3 0x0021dd6c in waitThread_ (m=0x1100360, initialCapability=0x0) at Schedule.c:2156 #4 0x0021dc50 in scheduleWaitThread (tso=0x13c0000, ret=0x0, initialCapability=0x0) at Schedule.c:2050 #5 0x00219548 in rts_evalLazyIO (p=0x29b47c, ret=0x0) at RtsAPI.c:459 #6 0x001e4768 in main (argc=2262116, argv=0x308548) at Main.c:104 I took waitThread_ as a clue and started digging deeper. Whenever I connect to the server or send a command I wait for X seconds and if not connected or desired command is not received I throw an exception which fails the script. I implemented the timeout combinator a couple of different ways, including that in the Asynchronous Exceptions paper but it did not help. I think the issue has to do with killing threads that are using FFI. Although I'm killing threads that call the Haskell connectTo, hGetBuf, etc. I think it's still FFI. I disposed of timeouts entirely, leaving connectTo as it is and using hWaitForInput on my socket handle to simulate timeouts. This improved things tremendously and I'm now able to run a few thousands of unbound script threads on Windows with OpenSSL FFI and everything. Memory usage is still higher than I would have liked and crashes in OpenSSL still happen when the number of threads/memory usage is really high so there's still room for improvement. I should probably go back to using a foreign finalizer (SSL_free) on the SSL descriptors rather than freeing them explicitly as the freeing does not happen if a script fails mid-way. I'm quite satisfied with my first Haskell project. I love Haskell and will continue hacking away with it. This list is invaluable in the depth of offered help whereas #haskell (IRC) is invaluable when speed matters. I'm quite amazed at the things I have been able to do, the expressiveness of Haskell and the clean looks. Clean looks can be deceptive, though, as they can hide code of amazing complexity. Fundeps, existential types, HList take a while to grasp. Also, I feel somewhat like a pioneer and I definitely got more than a fair share of arrows in my back. I had GHC run out of memory during compilation (fixed by SPJ), had it quit midway during compilation with an error about generated extents being too large in assembler code. I had GHC crash at runtime with an error like "fromJust not returning Just, this could not be happening!". Yesterday's error topped them all: internal error: update_fwd: unknown/strange object 0 Please report this as a bug to glasgow-haskell-bugs(a)haskell.org, or http://www.sourceforge.net/projects/ghc/ I think I got this when using +RTS -C0 -c. Overall, the experience with Haskell has been exhilarating and I'm already preparing to use it on my next projects like detecting collusion in poker as well as rake optimization (Dazzle paper very helpful here!). Still, I think that GHC can be a bit rough around the edges and I would think twice about writing high-performance network apps with it. Thanks, Joel P.S. The Glasgow Distributed Haskell (GdH) people are supposed to have a mailing list and I would love to share my findings twith them but I could not find the mailing list itself. -- http://wagerlabs.com/

2 2

RE: [Haskell-cafe] Re: IO exception not being caught
by Simon Peyton-Jones 17 Nov '05

17 Nov '05

Indeed! I always use braces and semicolons with do-notation. You are free to do so too! Nothing requires you to use layout. Indeed, you can freely mix the two. Simon | -----Original Message----- | From: haskell-cafe-bounces(a)haskell.org [mailto:haskell-cafe-bounces@haskell.org] On Behalf Of Joel | Reymont | Sent: 16 November 2005 15:37 | To: Haskell Cafe | Subject: [Haskell-cafe] Re: IO exception not being caught | | After almost two months with Haskell I'm starting to understand why | its use is not as widespread as... well pick a favorite language of | your own. My issue was that of indentation. | | Compare this working version: | | liftIOTrap io = | do mx <- liftIO (do x <- io | return (return x) | `catchError` | (\e -> do let x = fromIOError e | trace_ $ "Caught " ++ x | return $ throwError x | )) | mx | | With the one below | | On Nov 16, 2005, at 2:27 PM, Joel Reymont wrote: | | >> liftIOTrap io = | >> do mx <- liftIO (do x <- io | >> return (return x) | >> `catchError` | >> (\e -> return (throwError | >> (fromIOError e)))) | >> mx | | -- | http://wagerlabs.com/ | | | | | | _______________________________________________ | Haskell-Cafe mailing list | Haskell-Cafe(a)haskell.org | http://www.haskell.org/mailman/listinfo/haskell-cafe

7 7

Darcs and the Google Base
by Dimitry Golubovsky 17 Nov '05

17 Nov '05

This may be just funny, but... As the Google Base went live yesterday (11/16/2005), I tried to add the information about my HSFFIG project to the Base. As the Base allows to define arbitrary attributes (labels) for each item, I added the two of "Web URL" type: CABAL and DARCS holding urls for the project Cabal file and the Darcs repo respectively. I was turned down on "excessive capitalization" of these labels. I made them into "Cabal" and "Darcs". The former was accepted. the latter was turned down again as "misspelled". I wrote an "exemption request" explaining that Darcs is not a misspelling. However, it's been more than a whole day since my attempt, it hasn't been resolved. Interestingly, my item was published after I removed the "Darcs" label, but shortly after I resubmitted the exemption request for Darcs the item disappeared. -- Dimitry Golubovsky Anywhere on the Web

1 0

RE: [Haskell-cafe] Spurious program crashes
by Simon Marlow 17 Nov '05

17 Nov '05

On 16 November 2005 17:38, Joel Reymont wrote: > I'm getting crashes like this and I cannot figure out what the > problem is. I'm launching a bunch of threads that connect to a server > via TCP and exchange packets. > > I am running operations like connect and receive in a timeout > function that launches two threads and uses an MVar to figure out > who's done first. The timeout function then kills the two threads. > > Any ideas what could be causing this? I feel like a Haskell guinea > pig these days :-). I don't see any reason why you should be getting crashes here, but this is a delicate area (async exceptions + FFI). It's possible there's a bug, but as usual we need to reproduce the symptoms here. Can you help with a repro case? Regarding the behaviour of killThread, I believe the version in GHC is slightly different from the version described in the Asynchronous Exceptions paper, in particular the GHC version blocks until the exception has been delivered to the target thread (use another forkIO to get the fully async version). Cheers, Simon

2 2

ReaderT and concurrency
by Kurt Hutchinson 17 Nov '05

17 Nov '05

I'm writing a program that will be using multiple threads to handle network activity on multiple ports in a responsive way. The treads will all need access to some shared data, so I'm using an MVar. So far so good. The problem is that passing the MVar around everywhere is kind of a pain, so I was hoping to use a ReaderT monad on top of the IO monad to handle that for me. I have that working, but one piece seemed a bit out of place so I wondered if there was a better way. Below is a small test program that presents my question. My program uses forkIO to create the separate threads (Set A), and some of *those* threads will need to create threads (Set B). In order for the ReaderT to handle the environment of the threads in Set B, do I have to perform another runReaderT when forking? Or is there a way to get the ReaderT environment automatically carried over to the newly created Set B thread? See the "NOTE" comment in the code below for the particular spot I'm asking about. > import Control.Monad.Reader > import Control.Concurrent > > type MyReader a = ReaderT (MVar Integer) IO a > > main = do > mv <- newMVar 1 > forkIO $ runReaderT (printAndFork "one" 5) mv > forkIO $ runReaderT (printAndFork "two" 5) mv > getChar -- Pause so the threads can print before program exit > > printAndFork :: String -> Integer -> MyReader () > printAndFork _ 0 = return () > printAndFork str n = do > mv <- ask > lift $ do > modifyMVar_ mv $ \i -> do > print $ str ++ show i > return (i + 1) > -- NOTE: Is this runReaderT necessary to carry the > -- environment to the new thread? Is there a better way? > forkIO $ runReaderT (justPrint $ "inner " ++ str) mv > printAndFork str (n - 1) > > justPrint :: String -> MyReader () > justPrint str = do > mv <- ask > lift $ do > modifyMVar_ mv $ \i -> do > print $ str ++ show i > return (i + 1) > return () Kurt Hutchinson

6 7

RE: [Haskell-cafe] layout problem
by Simon Marlow 17 Nov '05

17 Nov '05

On 16 November 2005 17:15, Christian Maeder wrote: > Simon Peyton-Jones wrote: >> Indeed! I always use braces and semicolons with do-notation. You >> are free to do so too! Nothing requires you to use layout. Indeed, >> you can freely mix the two. > > I would not recommend braces and semicolons, because these allow a bad > layout (easy to parse for a compiler, but hard to read for a human), > unless you invest the time to make a tidy layout despite the braces > and semicolons. (So why not only make a tidy layout?) > > Surely, a different layout may change your semantics (in rare cases). > A missplaced "_ -> error ..." case usually causes a pattern warning. > >>>>> liftIOTrap io = >>>>> do mx <- liftIO (do x <- io >>>>> return (return x) >>>>> `catchError` >>>>> (\e -> return (throwError >>>>> (fromIOError >>>>> e)))) > > I'ld rather avoid the infix `catchError' and write: > > liftIO $ catchError > (do ... > ) $ \e -> I generally prefer to use the "handle" variants rather than "catch": liftIO $ handle my_handler $ do x <- io return (return x) where my_handler e = return (throwError (fromIOError e)) Don't be afraid to name things if it makes your code easier to read. Cheers, Simon

1 0

RE: [Haskell-cafe] Reducing # of context switches
by Simon Marlow 17 Nov '05

17 Nov '05

On 17 November 2005 00:17, Joel Reymont wrote: > The latest GHC docs mention that the -C option takes a seconds value > whereas prior docs mention microseconds. Which is it? > > Also, do I pass +RTS -Cxxx or is it just -C? It is in seconds, eg. +RTS -C0.5 for switches every half a second. I've just fixed and cleaned up the docs a bit. Cheers, Simon

1 0

updated_fwd: unknown/strange object 0
by Joel Reymont 16 Nov '05

16 Nov '05

internal error: update_fwd: unknown/strange object 0 Please report this as a bug to glasgow-haskell-bugs(a)haskell.org, or http://www.sourceforge.net/projects/ghc/ This happened when I tried running with +RTS -C0 -c I think. -- http://wagerlabs.com/

1 0