- Haskell-Cafe - Haskell.org

Re: [Haskell-cafe] Darcs apply error while getting the complete GHC tree
by Joel Reymont 17 Dec '05

17 Dec '05

I thought I did :-). I just didn't get to 7.1. My bad and apologies! On Dec 17, 2005, at 7:07 PM, wld wrote: > Well, RTBG (Building Guide) ! > When building from CVS (I think, from darcs repo too), > you should do: > > autoreconf > ./configure -- http://wagerlabs.com/

1 0

Haskell School of Expression?
by Creighton Hogg 17 Dec '05

17 Dec '05

Hi, Has anyone recently tried to install the graphics libraries included on http://haskell.org/soe/software.htm ? I've tried to make the libraries, but I get these errors. make: Entering directory `/home/wchogg/coding/haskell/GraphicsLib/lib/x11' ffihugs +G +L"X_stub_ffi.c" X.hs Warning: unknown toggle `G'; ignoring. Warning: unknown toggle `L'; ignoring. runhugs: Unable to initialise Hugs (Unrecognised option ) I honestly can't figure out what the incompatibility is? I'm using version 20050308 of Hugs

2 1

Barrier implementation
by Peter Eriksen 17 Dec '05

17 Dec '05

Greeting, Something is not working for me, and I could use some more eyes on this. What I'm trying to accomplish is to implement a simpel barrier for ten worker threads (id = 0..9) using STM. With or without the barrier, the program produces an unordered interleaving of the output from the workers. Here's what I get with the program below: $ ghc --make Main.lhs $ a.out 0134568027913457896012579026813423904671238455702468159367839684012570279134685049137825901642375689134057892610462578903156012389473268457910267801345923924567801304689235714013679458256702465913878... And here's what I get without the line "atomically $ barrier tv id": $ a.out 1249056782934567210845619720538461975203698469175203469850123485076912348579406123894625738942106381592740631859274163841092315768491302578416930728254169302785693024917853029640217390856490... The first run should've been something like: 012345678901234567890123456789012345... since each worker thread 0..9 should write its id out once per iteration, and the workers should iterate in sync. Here's the code: > module Main where > > import Control.Concurrent > import Control.Concurrent.STM > import System.Random > > worker :: Int -> TVar Int -> IO () > worker id tv = do > sleepingTime <- randomRIO (0, 50000) > threadDelay sleepingTime > putStr $ show id > atomically $ barrier tv id > worker id tv Each worker sleeps for some time, then outputs its id and waits at the barrier for all the other workers to finish their sleep+output. > barrier :: TVar Int -> Int -> STM () > barrier tv id = do > passed <- readTVar tv > if (passed `mod` 10 == id) > then writeTVar tv (passed+1) > else retry The barrier is simply a global variable, tv, which holds the number of times any worker passed the barrier. Now, a worker may only pass the barrier iff the worker with an id one less just passed, or else it should block. > main :: IO () > main = do > tv <- atomically $ newTVar 0 > for [0..9] $ \i -> forkIO $ worker i tv > threadDelay (10*10^6) > > for = flip mapM_ The main thread just initializes the pass counter, starts 10 worker threads, and waits for ten seconds. I'd like to hear some comments on the approach, and perhaps even some insight into why it doesn't work. Regards, Peter Eriksen

6 9

RE: Re[2]: [Haskell-cafe] Substring replacements
by Branimir Maksimovic 17 Dec '05

17 Dec '05

>From: Bulat Ziganshin <bulatz(a)HotPOP.com> >Reply-To: Bulat Ziganshin <bulatz(a)HotPOP.com> >To: "Branimir Maksimovic" <bmaxa(a)hotmail.com> >CC: daniel.is.fischer(a)web.de, Haskell-Cafe(a)haskell.org >Subject: Re[2]: [Haskell-cafe] Substring replacements >Date: Fri, 16 Dec 2005 16:51:32 +0300 > >Hello Branimir, > >Friday, December 16, 2005, 5:36:47 AM, you wrote: >BM> I've also performed tests on dual Xeon linux box and results are > >just to let you know - GHC don't uses pentium4 hyperthreading, >multiple cpus or multiple cores in these tests Oh yes it does. I clearly see multiple threads that take unequal percentage on each virtual CPU. I guess that other thread is garbage collector thread. In case of hyperthreading, speed is gained by reduced memory latency by 30-60 % > >only way to make ghc using multiple processors is to use 6.5 beta >version, compile with "-smp" and explicitly fork several threads This is not the case as I see. On windows search replace test programs spawn 3 threads and on linux I'm not sure, but I've checked program that calls Haskell from C++ and GHC spawns additional thread, which is not my thread, that also performs something constantly, and I didn't spawn any thread from Haskell. So hyperthreading helps as it helps to optimise when several threads accesses memory as is in this test case. I can't see any other explanation why KMP search is slower on AMD 20% , but faster on Intel 30%, then straightforward search with my test. Greetings, Bane. _________________________________________________________________ Express yourself instantly with MSN Messenger! Download today it's FREE! http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/

1 0

Optimizing a high-traffic network architecture
by Joel Reymont 17 Dec '05

17 Dec '05

Folks, In my current architecture I launch a two threads per socket where the socket reader places results in a TMVar and the socket writer takes input from a TChan. I also have the worker thread the does the bulk of packet processing and a timer thread. The time thread sleeps for a few minutes and exits after posting a timeout event if it hasn't been killed before. My goal is to launch poker 2,000 bots that join the server "lobby" and sit there sending small keep-alive packets every few minutes. The ultimate goal is for 4,000 bots to be playing but I'm taking it one step at a time. This is Mac OSX Tiger with a couple of header files modified to allow FD_SETSIZE of 10240. This is the maximum allowed by 'ulimit -n'. I'm also running ghc 6.4.1, compiled after FD_SETSIZE has been increased. I can get to 2k bots without any trouble if I use a keep-alive timeout of 9 minutes. Memory usage with 2k bots is 161Mb of physical memory and 262Mb VM. CPU usage 20-40%. Memory usage is constant once all bots have been launched. With a 1 minute keep-alive timeout system is starting to get stressed almost right away. There's verbose logging going on and almost every event/packet sent and received is traced. The extra logging of the timeout events probably adds to the stress and so, I assume, do the extra packets. New bots are being launched very slowly even with just 200 bots already running. Based on the above, would you have any suggestions for an improved architecture? I will try 1) disabling logging alltogether and 2) increase thread stack size to 3k (+RTS -k3k) as per Simon Marlow's suggestion. As per simon if a thread stack space is between 2k and 4k then each thread gets its own memory block (right Simon?) and threads are not GCd then. I'm a bit concerned about trippling my memory use with -k3k, though. I'm not sure if switching to a continuations-based framework will help me. Has anyone tried this? Thanks, Joel -- http://wagerlabs.com/

6 27

RE: [Haskell-cafe] Battling laziness
by Simon Marlow 16 Dec '05

16 Dec '05

On 16 December 2005 15:23, Joel Reymont wrote: > Looking at http://wagerlabs.com/randomplay.hd.ps I see closures > (constructors?) in this order > > <Script.Array.sat_s46N> > W8# > I# > <Script.Array.fromIntegral_s453> > <Script.Endian.sat_s1WxM> >> > <Script.Endian.sat_s1WF2> > W16# > <Script.PicklePlus.sat_s38YS> > stg_ap_2_upd_info Ok, so your heap is mainly full of (a) thunks generated by something in Script.Array, (b) Word8s, and (c) Ints. > This tells me it's something having to do with array code. I'm > attaching the Script.Array module at the end. This report does not > tell me who is retaining the data, though. > > Looking at http://wagerlabs.com/randomplay.hy.ps I see types ordered > like this > > * > Word8 > Int > ->* > [] > Char > Word16 > TableInfo interesting... Word8 and Int correspond to the -hd output above, but '*' indicates that the type of the <Script.Array.sat_s46N> is polymorphic. Completely polymorphic closures like this are usually (error "something"), which is a silly thing to fill up your heap with :-) I'm a bit mystified though, because looking at the code for Script.Array, all your arrays are unboxed, so I don't know where all the Word8s and Ints are coming from. It might be useful to do "+RTS -hyWord8 -hc" to see who generated the Word8s. Oh, and it looks like you aren't doing -auto-all, that would probably be helpful. Cheers, Simon

2 5

RE: [Haskell-cafe] Optimizing a high-traffic network architecture
by Simon Marlow 16 Dec '05

16 Dec '05

On 16 December 2005 15:19, Lennart Augustsson wrote: > John Meacham wrote: >> On Thu, Dec 15, 2005 at 02:02:02PM -0000, Simon Marlow wrote: >> >>> With 2k connections the overhead of select() is going to start to >>> be a problem. You would notice the system time going up. >>> -threaded may help with this, because it calls select() less often. >> >> >> we should be using /dev/poll on systems that support it. > > And kqueue for systems that support that. Much, much more efficient > than select. Yeah, yeah. We know. We just haven't got around to doing anything about it :-( It's actually quite fiddly to hook this up to Handles - see Einar's implementation in Network.Alt for instance. Cheers, Simon (who wished he hadn't mentioned select() again)

1 0

RE: [Haskell-cafe] Battling laziness
by Simon Marlow 16 Dec '05

16 Dec '05

On 16 December 2005 12:42, Joel Reymont wrote: > On Dec 16, 2005, at 12:36 PM, Simon Marlow wrote: > >> If script#9 is the cost center attached to all of your leaking heap >> data, then you're already a long way to finding the problem. It'll >> help even more to find out whether it is just unevaluated copies of >> "takeEmptySeat Holdem affid []", or something else (-hd, -hy will >> help here). Try +RTS -hy -hcscript#9, for example. >> >> One obvious thing to try is replacing the '$' before {-# SCC >> "script#9" #-} with '$!'. And similarly in takeEmptySeat. > > Let me try these and report my findings. > >> (I should say that we definitely plan to update these for STM, but >> it's not completely trivial (I checked). Volunteers definitely >> welcome). > > I volunteer! Just need some pointers on where to get started. I learn > quickly but need to be guided ;-). Plus, I need this the most, right? I was slightly mistaken: lag/drag/void profiling is pretty easy. Take a look at ghc/rts/LdvProfile.c and add relevant cases for STM objects to processHeapClosureForDead(). If you fix this up and test it we should be able to get it into 6.4.2. Retainer profiling is much harder; the code is in RetainerProfile.c/RetainerSet.c. Cheers, Simon

2 2

RE: [Haskell-cafe] Optimizing a high-traffic network architecture
by Simon Marlow 16 Dec '05

16 Dec '05

On 15 December 2005 10:21, Joel Reymont wrote: > Here are statistics that I gathered. I'm almost done modifying the > program to use 1 timer thread instead of 1 per bot as well as writing > to the socket from the writer thread. This should reduce the number > of threads from 6k (2k x 3) to 2k plus change. > > It appears that +RTS -k3k does make a difference. As per Simon, 2-4k > avoids the thread being garbage collected because each thread gets > its own block in the storage manager. Simon, did I get that right? > > BTW, how does garbage-collecting a thread works in this scenario? My > threads are very long-running. > > The total is the number of bots launched, lobby is how many bots > connected to the lobby. Failed is mostly due to connection reset by > peer errors. The Windows C++ server uses IOCP and running a firewall > was apparently interfering with that somehow. I hate Windows :-(. > > --- Test#1 +RTS -k3k as per Simon. Keep-alive timeout of 9 minutes. > > Total: 1961, Lobby: 1961, Failed: 0 > Total: 2000, Lobby: 2000, Failed: 1 > > This test went smoothly and got to 2k connections very quickly. Maybe > within 30 minutes or so. I did not gather CPU usage, etc. statistics. > > --- Test #2, No thread stack increase, 1 minute keep-alive timeout, > more network traffic > > With a 1 minute timeout things run veeery slow. 86 physical and 158Mb > of VM with 1k bots, CPU 50-60%. Data sent/received is 60-70 packets > and 6-7kb/sec. Killed after a while. > > The statistics are phys/VM, CPU usage in % and #packets/transfer speed > > Total: 1345, Lobby: 1326, Failed: 0, 102/184, 50%, 90/8kb > Total: 1395, Lobby: 1367, Failed: 2 > Total: 1421, Lobby: 1394, Failed: 4 > Total: 1490, Lobby: 1463, Failed: 4, 108/194, 50%, 110/11Kb > Total: 1574, Lobby: 1546, Failed: 4, 113/202, 50%, 116/11kb Hmm, your machine is spending 50% of its time doing nothing, and the network traffic is very low. I wouldn't expect 2k connections to pose any problem at all, so further investigation is definitely required. With 2k connections the overhead of select() is going to start to be a problem. You would notice the system time going up. -threaded may help with this, because it calls select() less often. If that's not the cause, we should find out what your app is doing while it's idle. If there are runnable threads (eg. the lauchner), then the app should not be spending any of its time idle. Cheers, Simon

9 10

RE: [Haskell-cafe] Battling laziness
by Simon Marlow 16 Dec '05

16 Dec '05

On 16 December 2005 12:08, Joel Reymont wrote: > -hc points to script#9 below. > > script (_, _, affid) (Custom (JoinedTable 0)) = > do {-# SCC "script#8" #-}push "takeEmptySeat" $ > {-# SCC "script#9" #-}takeEmptySeat Holdem affid [] > {-# SCC "script#10" #-}return $ Eat $ Just Go > > What takeEmptySeat does it call pickTable > > takeEmptySeat game_type _ filters Go = > do push "pickTable" $ pickTable game_type filters > return $ Eat $ Just Go It's hard to pick out the cause of a space leak from just a fragment of the program, but I'll try to give you some pointers. If script#9 is the cost center attached to all of your leaking heap data, then you're already a long way to finding the problem. It'll help even more to find out whether it is just unevaluated copies of "takeEmptySeat Holdem affid []", or something else (-hd, -hy will help here). Try +RTS -hy -hcscript#9, for example. One obvious thing to try is replacing the '$' before {-# SCC "script#9" #-} with '$!'. And similarly in takeEmptySeat. > Overall, -hc does not help me figure out where my data is being > retained. My understanding is that I need to do -hbdrag,void fo > rthat. I did not try -hd and -hy, they would only help me narrow down > the producers, right? Not necessarily; lag/drag/void only tells you about certain kinds of space leaks. It's another tool in the box, and quite often you can get away without it. Retainer profiling similarly. (I should say that we definitely plan to update these for STM, but it's not completely trivial (I checked). Volunteers definitely welcome). > My program seems to spend 70% of the time collecting garbage. Notice > the HUGE overall allocations. This is my trying to launch 4k bots > over 8 hours. Only 1k bots were launched and just 300 of those got to > play. Maybe because they did not have time with all the garbage > collection :-). Note that your GC time is inflated quite a bit due to profiling (it makes every object larger). The plan to reduce GC time is, in this order: squash space leaks, reduce allocation (to reduce GC load), and then tweak GC parameters. Cheers, Simon

2 2